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Introduction and Overview 

This report responds to a request from the Division of Research and 
Development Resources (DRDR) of the U* S* Office of Education’s National 
Center for Educational Research and Development* DRDR was seeking 
a design for evaluation that will enable it to make responsible decisions 
regarding institutions and programs it supports. The authors of 
the report met August 5 through 9, 1971 to analyze DRDR's request and 
to prepare the report. 

The evaluation system developed in response to the request 
meets five major DRDR specifications. First, it focuses on the ten 
decisions explicated in DRDR* $ New Support Policy (Frye 1971). Second, 
it responds to five specified problems, Third, it presents an 
organizational framework within which the proposed evaluation system 
functions. Fourth, it projects the procedures to follow In implementing 
the new system. And fifth, it supplies a set of criteria and related 
guidelines to use in applying the proposed evaluation approach to the 



tan specified decisions. This chapter, then, provides overviews 
associated with these five DRDR specifications. 

Maturity Model and the Ten Evaluation Types 



When this Project Group met to respond to DRDR's charge, the members 
reviewed the new Institutional policy of the division as summarized 
in the working paper entitled "DM! Institutional Support and Evaluation 
Policy, 11 dated 3 June 1971- The paper posited a "maturity model" that 

f 

Included ten decision-points calling for evaluation, 

\ 

1, Award of grants for planning new institutions . Planning- 
grant proposals will be solicited from groups and agencies interested 
in founding new educational institutions. Such a planning-grant 
proposal will be evaluated prior to a funding decision. 

The key question to be answered in this evaluation 
is: Should the request for a planning grant for a 

new institution be funded? 

2. Award of operational grant for starting new institutions . 
The award of a planning grant will lead to the development of a 
full-scale institutional proposal. The latter will be evaluated 
between three and six months after the termination of the planning 
period. If accepted, the proposal will provide funds to establish a 
new institution. 

The key question is; Should a given proposal to 
start a new institution be funded? 
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3- Approval of a new institution program plan . After a 
development period of one to two years, the new institution will be 
required to submit a report summarizing its organization and defining 
its mission. At least one program plan must be developed. 

The key question Is: Is the basic program plan 

of a given institution satisfactory? 

4, Certif i cating a new institution^ management capabilities . 
During this phase the new Institution will be requi red to bring one or 
more program plans into full operation. After approximately two years 
a report will be required to determine if the Institution is mature 
enough to continue with program support. 

The key question Is: Is the institution capable of 

managing Its basic program plan? 

5, Award of grants for planning new programs . An 
Institution that has successfully completed its maturity review will 
be classed as a "mature Institution'’ and will be eligible to apply 
for one or more development grants for new programs. (Already 
mature institutions may submit development grant-p roposa 1 s without 
going through decision points 1-4.) The proposals for such 
program-development grants will be evaluated. 

The key question is: Should a given developmental 

grant for a new program be funded? 
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6* Approval of Program Plans , The award of a new program- 
development grant will result in the planning of a new program that 
will be evaluated. 

The key question is: Should the Office of 

Education commit itself to support the proposed 
program plan? 

7, Milestone Rev i cw . A funded program will have specified 
milestones at which certain phases of the program work must be 
completed. As these milestones are reached, a report will be submitted 
for evaluation. This evaluation, unlike those at earlier or later 
stages, will be made by a review team appointed by the Institution 
rather than by OE; however, the Information collected by the team 

will be shared with DRDR. 

The key question is: Are significant milestones 

In the program plans being reached, on time, and with 
sat i sfactory quality? 

8. Modification of DRDR program Support for Mature 
Institution Motivated by Special Circumstances . Special 
circumstances, e.g., the departure of key leadership personnel, 

that appear capable of altering the i nst i tut \ on 1 s capability to continue 
its programs, may prompt DRDR to investigate. A special report, 
relating to the circumstances, may be required for evaluation by DRDR. 
The key question is: Do current circumstances 

of the programs or Institution require redirection 
or changes in OE support? 
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3. Program Outcome Evaluation . When a program concludes, 
its final outcome will be evaluated. 

The key question is: Did the program reach 

its objective? 

10* Budget and Contract Review , This review differs from 
the previous nine because it occurs annually as a function of the HEW 
funding cycle. Its purpose is to determine whether or not adjustments 
should or must be made in the institutions funding pattern* 

The key question is: With given funds each 

year, how should any given program be adjusted, 
stretched, or compressed? 

Decision points 1 and 5 relate to planning activities; 
decision points 2 and 4 to Institutional factors (initiate, adjust); 
decision points 3, 6, 7? 8, and 9 to program factors (initiate, adjust, 
terminate); and decision point 10 to annua 1 -budget i ng factors. 

Decision points 4, 6, and 8 probably are the most critical,, 

Decision points 1, 2, 3? and 5 require only minimal evaluation 
activity by DRDR and 10 is part of the annual budgeting process. 



Claims for the Proposed Evaluation System 

The authors of this report contend that their proposed evaluation system, 
together with the new DRDR Support Policy, comprises a sound strategy 
for overcoming five problems that plague attempts to evaluate and make 
decisions about institutions and their associated programs. 
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Presently, it is difficult to recoup and redirect Office of 
Education funds. Consequently, new institutions or programs to address 
emergent or new problem areas are unavoidably denied. The new 
evaluation and decision-making system will overcome this problem by 
determining early whether an institution is a good risk to conduct 
sound research and development work - by setting milestones, by 
requiring final evaluation for programs, and by Insuring that milestones 
and terminal evaluations are regarded with importance. 

The second problem involves evaluations. They often lack 
the credibility needed to support decisions to terminate programs or 
institutional support* The proposed evaluation system attempts to 
overcome this by providing for (1) a careful selection of judges; 

(2) judges trained to make their evaluation based on data produced 
from institutions and programs; (3) three or five judges who make 
independent judgments in reference to the same set of data; and (4) 
an audit of evaluation procedures and results that provides for the 
def ens i b i 1 i ty of decisions based on those evaluations. 

A third problem has resulted because of 0E‘ s inability to 
rank institutions and/or their programs. The proposed system provides 
explicitly for a rank ordering of institutions and programs according 
to a psychometric procedure to be administered by an external technical 
se rvi ce agency. 

A fourth problem involves the confusion regarding nurturent 
and evaluative roles with respect to administering institutions 1 
programs. The proposed system distinguishes sharply between these 
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roles and calls for a separate evaluation office that administers 
the evaluative roles. 

The fifth problem Is the ill-defined criteria for evaluating 
institutions and programs. This report lists well explicated criteria. 
And the report specifies which of the criteria applies to which of 
the decision situations. Also, the criteria are presented in the 
appendix in the form of sped f i c gu i del i nes for evaluations associated 
with each of the 10 decisions. 

Generally, then, the evaluation system proposed responds to 
DRDR 1 s charge. 

Overview of the Organ i zat i onal Framework for the Evaluation System 

It is assumed that the implementation of the proposed evaluation designs 
depends on the following seven classes of personnel: 

0E decision-makers . Certain key 0E personnel have the 
legal responsibility to propose decisions contingent on each of the 
ten evaluations. These decisions, which must be serviced by 
evaluation Information, are best reflected in the ten questions posed 
for each of the evaluation points described above. Positive answers 
to the questions result in new or continued funding; negative answers 
result in Institutional or program recycling, or termination of funding, 
DRDR evaluators . The decisions to be made are serviced by 
evaluation information, it is the responsibility of DRDR evaluators to 
manage the evaluation and to interpret and report their findings to the 






DRDR dec i s i on-makers. (Much of the evaluation Information, 

Incidentally, is generated by the institutions being evaluated* outside 
judges, and by analysis performed by outside agencies.) 

Institutional decision-makers . A variety of decisions regarding 
program management will be made at the institutional level by 
institutional decision-makers. Planning decisions (e, g, , what needs, 
problems, or opportun I t i es shall become the focus of an institutional 
program?), structuring decisions (e. g, , how shall a given need, 
problem, or opportunity be addressed?), implementing decisions 
(e. g, , are the means projected actually in use and being applied 
according to specifications?), and recycling decisions (e.g. , should 
a program component be recycled or terminated?) must be made and 
each must be serviced by an i nst i tut lonal evaluation mechanism. 

Institutional evaluators . Large, local evaluation staffs will 
be required to provide institutional dec i s i on-makers with a variety 
of evaluative information. These local staffs also must service 
certain DRDR needs by collecting other information specifically required 
by DRDR and/or by aggregating local data for DRDR purposes. 

Expert judges , DRDR will not have an evaluation staff of 
sufficient size to perform all aspects of the evaluations; moreover, 
such an eva 1 uat i on , even if possible, wou Id la ck c red T b ! 1 ity both with 
the institutions being evaluated and with key outside reference 
groups (e, g. , USQE, HEW, the Congress, and concerned publics). Hence, 
arrangements will have to be made to provide expert judges who can 
participate independently in the evaluation process. Such judges will 
be required for two purposes: to evaluate the proposals and/or reports 
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submitted for various evaluation purposes (the ten decision points) 
and to verify and extend the information contained ?n such proposals 
and/or reports through site visits* Site visits may not be needed at 
decision points 1 , 2, 5, 7, and 10 (although an i nst i tut i ona 1 1 y sponsored 
site visit may occur at point 7). 

Contract Service Agents , Certain services requisite to 
evaluation performance will have to be provided by outside service 
agencies or contractors. For example* consultants may be needed to 
engage in continuous study* refinement* and extension of the program 
guidelines; computer services will be required for certain data-ana lysis 
tasks; analyses concerned with determining the marketability of products 
resulting from funded programs may be needed; and spec? al I st s may be 
required to assist in personnel training. 

Perhaps the major outside service required will be an anal ys is 
agent to assist in ranking institutional proposals and reports for 
quality. As will be noted briefly In the following chapter and described 
in detail in chapter three, proposals will be given comparative 
rankings for quality at two different points in the review process. 

These rankings will be developed by the application of psychometric 
techniques dealing with comparative judgments. This task will be 
difficult to accomplish without the assistance of an outside contractor 
having the necessary technical expertise and computer soph i st i cat i on . 

Eva 1 u i at i on aud ? tors . Despite all efforts to maintain integrity, 
an evaluation carried out solely by employees of DRDR, the local 
Institutions, and their agents may lack credibility with certain key 
audiences, c. g. , the Congress. To provide such credibility, evaluation 
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auditors — nationally known figures whose integrity is beyond question 
-« will be employed to audit, on a sampling basis, evaluations of each 

t 

of the ten types. In addition, they will attest both to the soundness 
of the procedures employed and the validity, reliability, and objectivity 
of the conclusions generated, 

A fuller explication of the organizational framework can be 
found in chapter six. 

Overview of the Proposed Model for Evaluation and Deci s i on-Maki ng 

The proposed evaluation designs are all based (with minor variations) 
on a process model involving DRDR and institutional evaluators, expert 
judges, and evaluation auditors, as follows: 

-- Guidelines for proposals and reports, as required in the 
ten evaluations, are distributed by DRDR, 

Institutional decision-makers, supported by their evaluation 
staffs, prepare the required reports and/or proposals. 

-- DRDR decision-makers establish panels of expert judges and 
evaluation auditors that can be assigned, on a systematic basis, to 
particular evaluations, 

-- Proposals or reports are received by DRDR evaluators who 
give them an initial check to ascertain that all required information has 
been included. Documents failing this test are recycled to the institution 
f o r rev i s i on, 

-- The factual information contained in the reports and/or 
proposals is checked for validity by a DRDR evaluator with some 
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experience with the institution in question. The evaluator notes 
discrepancies for later referral to site reviewers who can make an 
onsite verification. Alternatively, in less important cases, the 
evaluator may make a verification site visit himself. 

— proposals and/or reports are submitted to a group of 
independent judges who independently evaluate the documents On the 
basis of criteria specific to the decision type (see chapter two for 
an overview of these criteria). Documents failing this test are 
rej ected, 

— Proposals and/or reports passing the expert-judge check 
are ranked by the analysis agent on the basis of a metric relating to 
their relative quality. (This metric, explicated in a later chapter, 
depends on a method of comparative judgments drawn from psychometric 
theory. ) 

-- Proposals and/or reports ranked "sufficiently high" on 
the preceding step are resubmitted to the expert judges that have 
been transformed into site reviewers. A further onsite evaluation is 
made in terms of criteria identical to those utilized for the reading 
review. Proposals and/or reports failing this step are rejected. 

(Note: This onsite evaluation does not occur in decision points 1 , 5 , 

7, and 10, although decision 7 involves a site visit by a panel of experts 

appointed by the institution.) 

Institutional requests passing the site review check are 

ranked by the analysis agent on the basis of relative quality using the 
same metric identified above* 
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-- Institutional requests ranked “sufficiently high 11 on 
the preceding step are approved for funding. 

-- The entire process summarized above is monitored on a 
sampling (p ronaba 1 i st i c) basis by an evaluation auditor who can attest 
to its validity, reliability, and objectivity* 

A fuller explication of the model for evaluation and decision- 
making can be found In chapter two* Exemplar applications for decision 
types 2, h , 6 S and 9 can be found in chapter three* 

Overview of Criteria To Be Applied in the Evaluation of DRDR Programs 

Each of the ten evaluation types posited in this paper services a 
decision point at which certain institutional or p rogram- rel ated decisions 
are made* It is clear that these decisions cannot be made in the 
absence of specific criteria; consequently, one of the Advocate Team's 
assigned tasks was to generate a listing of criteria for this purposes 
By an iterative process, 357 candidates for criterion status 
nominated by this Advocate Team were reduced to a taxonomy of ten major 
categor i as* 

-- Significance of focus (institutional mission or program 

goal ) 

-- Significance of anticipated outcomes (demonstrated 
improvement In reading ability of d I sadvantaged five-year olds) 

-- Uniqueness (concentration of r &- d on neglected educational 
problems or concentration on solution to geographically specific problems) 
-- Viability (institutional or program) 
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-- Planning and evaluation adequacy 

“ Program balance (between research and development) 

Adequacy of program elements (e. g. , objectives, procedural 
design, and app rop r i ateness of personnel) 

Cost (appropr i ateness of personnel and time allocations) 

-- Economic efficiency (cost~benef i t considerations) 

-- Potential for out s i de support (e. g„ , service agencies, 
other funding sources) 

Each of these categories is broken into several subcategories 
(See chapter four,) 

Inspection of the categories indicated two basic dimensions 
along which it was useful to array the criteria: relevance to 

j ns t j tut ions , on the one hand, as against relevance to p ro grams 3 on 
the other; and relevance to the future (prospective view, based on 
proposals), on the one hand, and relevance to the past (retrospective 
view, based on reports), on the other. Using these two dimensions, 
four sets of criteria relating to each of the four types were arranged: 
prospective institutions, prospective programs, retrospective institutions, 
and retrospective programs. Moreover, it was possible to relate each 
of the ten types of evaluation (except number 10, that is mixed) to 
one of these four criterion types. 

The complete array of criteria appears in chapter four. The 
criteria are applied In guidelines, to be used by DRDR, at the end 
of this report. 

The remainder of this report explicates and illustrates the 
evaluation system. Chapter two presents the basic model for applying 
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the evaluative er I tor i a; chapter three illustrates how the model 
applies to decisions 2, 4, 6, 7 f and 9; chapter four presents and 
analyzes criteria to be used in applying the proposed evaluation 
system to the ten specified decisions* 

Chapter five includes four sets of special procedures* The 
first suggests procedures for selection, training, and assignment of 
expert judges. The second recommends instrumentation and quantitative 
procedures f or* obtaining and analyzing comparative judgments* The 
third presents procedures for site reviews, and the fourth gives 
recommendat i ons for the evaluation audit* 

Chapter six, a description of the evaluation system, considers 
institutional roles, suggestions for organizing DRDR, and a discussion 
of funding of the proposed evaluation system* Appendix A provides 
guidelines to be used by institutions and judges in preparing the basic 
data needed to support the Maturity Model's ten decisions. 

The final chapter provides recommendations* An overall 
chart depicting the relationship between selected factors in the 
proposed evaluation system and the ten support policy decisions is 
located in appendix B, This chart should be folded out at this point 
for ease of reference during the reading of the body of the report. 
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A General System Model for Application of the Criteria 

This chapter describes the system model devised to service the ten 
support policy decisions. The model applies to all ten of the 
decisions, but it is responsive to the idiosyncrasies of each. And 
it is useful to DRDR evaluators and decision-makers because it 
adheres specifically to the new DRDR support policy. 

Evaluation Requirements of the Ten Decisions 

Examination of the ten decisions reveals different kinds of primary 
concerns. Decisions 1, 2, and k primarily address questions about 
an institution; e.g., its management and personnel, with lesser 
attention given to its program development activities. Decisions 
3, 5, 6, 7, 8, and 9 primarily concern programs and their development, 
with somewhat less attention given to the institution and its 
management capabilities. In these latter decision situations the 
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management is considered only as it affects or is perceived to be 
the cause of the difficulty in program development* Of course 
decision situation ten differs from the others because it concerns 
an annual budget review conducted by the DRDR program monitor. 

Other differences also are apparent* Some of the decisions 
are more important (e.g,, 2, 3> 4 f 6, and 8) because they result 
in more significant commitments to funding. Another difference 
concerns the complexity of the decision and the data base required 
to make It* For example, decisions 1, 2, 3, and 5 require 
considerably less information and judgment than do the others* And 
decisions 7 and 10 require less effort on the part of DRDR staff 
than do the others. However, in the case of decision 10, a funding 
crisis at the USOE level may force serious cuts to be made in DRDR- 
funded programs at the time of annual funding decisions* If the 
crisis occurs, the evaluation unit will draw information about 
milestone reviews from its files and supply it to the division's 
decision-makers. Similarly, entries in the files regarding 
intervention reviews will be gathered and summarized for the 
decision-makers. Information on the success or failure of programs 
to reach their milestones and on the ability of institutions to 
weather Intervention reviews will be the influencing factor 
determining the allocation of the limited funds for the year ahead. 

Decisions 4, 6, and 8 are the most complex and imply 
greater commitments than do the others; therefore, the system calls 
for five judges to review the documents and to make site visits. 
Decisions 2 and 3 (planning grant review and new program-plan 
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review) are next in terms of complexity and commitments; therefore, 
three judges to review documents and to make site visits are proposed* 
decision 9? which concerns the outcomes and attainment of objectives 
of a program, requires three judges to review documents that give 
evidence of attaining objectives* However, no site review is needed* 

A three-member judge panel is proposed for review of documents for 
decisions 1 and 5 that entail planning grants. 



General System Model 



The system depends on interested parties or institutions furnishing 
information to DRDR in response to clearly understood guidelines. 
DRDR ? after checking the content, will utilize expert judges to 
validate facts and rate quality* These assessments, then, will be 
submitted to an analysis system performed by an outside contractor 
to rank order the results and/or proposals* 



The flow chart of the model (on page 19) and subsequent step- 
by-step description cover all decision situations in the system. For 
the more complex decisions requiring site visits, the full model Is 
used. For other decisions, where no site visits are projected, 
only the first two stages of the model are utilized. 




The model calls for three stages in the decision process* 
The first stage (1,0-1. 3) entails a content check by the DRDR 
evaluator and a recycling (if necessary) for more information* 

The second state (2.0-2.11) utilizes a systematic procedure to 
select and assign judges to read documents and make assessments. 
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The assessments are then utilized in an analysis system to rank 
order the results and to make decisions based on their relative 
quality- For decisions 1, 5i and 9 the process terminates and 
commitments are made at this point- For other decisions (2, 3 ? 

6, and 8) the process continues through stage 3 (3-Q-3-9)* During 
the third stage the expert judges, assigned to assess the quality of 
written documents, make site visits and further validate and assess 
the quality of the proposals. These assessments then are submitted 
by contract to an analysis system that results in a rank order for 
each decision situation. St rat ? f icat ions within them occur where 
different programs are involved. Again, a rank ordering of relative 
quality is determined and becomes the basis for decisions about 
further commitments. 

Although slight modifications are required for decision 7 
- where the mature institution conducts its own program-milestone 
review and reports the results to DRDR - and decision 10 - where 
DRDR conducts a budget review and adjustment - the system is adequate 
to cover needed information, assessment, and analysis to make the 
ten decisions indicated in the mature Institution policy. 



Description of Flow Chart 
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Following Is a step-by-step explanation of the system model flow- 
chart ; 

1.0 Distribute gui del i nes - Guidelines are developed 
based on criteria for the decision situation and notice of 
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avai lability is sent to interested parties along with a date for 
response to guidelines (if appropriate). Guidelines then are sent 
to the parties requesting them. (Prototype guidelines appear in 
appendix A.) 

1.1 Rece i ve repo r t $ or proposals . Proposals or reports 
come to the PRDR monitor where they are numbered consecutively or 
otherwise identified consecutively on a list, 

1.2 Make decision A . The DRDR evaluator checks with 
guidelines to determine if content of each proposal is sufficient 
(only presence or absence of required information Is noted). If 
required information is present in the proposal or report* the 
report goes to section 2.2. If information is missing* the report 
is returned to the group or institution, 

1.3 Return for revision . Proposal or report is returned 
with missing Information noted. Institution or group has one week 
to revise and resubmit the document, 

2*0 Identify expert judges . It is assumed that expert 
judges have been identified and classified Into different areas of 
expertise. From this pool* DRDR officials Identify expert judges 
according to a procedural plan (specified in Chapter V) . The 
appropriate number of judges (depending on the number and nature 
of decisions to be made) Is identified (as specified elsewhere) and 
the DRDR evaluator determines their availability for both reading 
proposals or reports and for making a site visit if necessary. 

2.1 Assign judges . DRDR assigns judges according to a 
procedural plan (specified in Chapter V), 
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2.2 Verify authenticity . The DRDR evaluator considers 
factual information in the report or proposal and verifies its 
accuracy. This may include discussion with the DRDR program monitor, 
phone calls to the institution or group concerned, or a site visit. 

If the decision to be made calls for a follow-up site review, then 
any questionable information may simply be noted and held for the 
site reviewers. However, if no such follow-up site review is 
intended, the evaluator must make an effort to validate the 
information involved - even if it requires him to conduct a site 

visit. 

2.3 Refer discrepancies to site visitors . Any 
questionable information will be noted and held aside by the DRDR 
evaluator to call to the attention of the site reviewers. 

2.4 Distribute reports to judges . The DRDR evaluator 
sends each reader four items, 

a. The report or proposal 

b. Guidelines (criteria) for judging the report 
or proposals (see appendix A) 

c. Instruments (Guidelines to serve as the basis 
for Instruments are included in the appendix A.) 

d. Instructions for completing the instrument and the 
date for returning It 

2.5 Receive assessments from judges . Assessments received 
from the judges are checked and analyzed by the DRDR evaluator. 

2.6 Send data for analysis . Instruments are organized 
according to stratifications, as explained in procedures for analysis. 
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Contractor sends 



and sent to the contract service agent. 

2* 7 Receive analyses from contractor , 
analyses to DRDR evaluator, who organizes and checks to see if all 
are returned. 

2 -8 Summarize analyses (ranks) . The DRDR evaluator 
summarizes the rank orderings within each stratification and prepares 
them for the Internal review. 

2.9 Perform internal review , DRDR decision-makers meet 
and review the data presented by the evaluator on each set of 
decisions within each stratification cluster. Policies and 
constraints are considered in preparation for making the decisions, 

2 -l° Make decision B , Decisions are made concerning each 
decision situation and rejected proposals or reports are classified 
and col I ected , 

2*11 Re j ect . DRDR dec? s I on-makers notify groups or 
Institutions of the rejection of their proposal, 

3.0 Site review required . Proposals or reports that 
survive are studied to determine the necessity of a site review. This 
Is the only place that indicates site reviews are prompted as opposed 
to being automatic. Those requiring a site review are prepared to go 
to 3.1* Those not requiring further review go to 0 and the successful 
groups or institutions are notified of the funding decision. 

3-1 Assign site visitors . DRDR assigns site visitors, 
previously used as readers, to visit the appropriate sites. The 
judges who reviewed an Institution's report or proposal are assigned 
to the same institution for site visits. The evaluation monitor sends 
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the reviewers (a) the report, (b) the reviewers 1 questions, (c) the 
instruments, and (d) the Instructions for use of Instruments and dates 
of site review. The Institutions are sent (a) criteria, (b) questions 
that site visitors will be using, (c) names of the site visitors, 

(d) dates of review, and (e) date when a decision will be reached, 

3.2 Conduct site review , DRDR evaluators assist in the 
site review according to the procedure (described in chapter five), 

3.3 Receive assessments from judges . The assessments 
from judges who made site visits are received and made available 
to the DRDR evaluator, who organizes them according to decision 
situations and stratifications within decision situations. If 
necessary (e. g. , program plans of a similar substantive nature)* 

3.4 Send data for analysis . After analyses are returned, 
they are prepared for review by the DRDR evaluators. Rank orderings 
within each category are summarized and prepared for the internal 
rev I ew. 

3.6 Per f o rm internal review „ DRDR decision-makers meet 
with evaluators to study the data on rankings and to consider 
priorities and constraints in preparation for making decisions* 

3.7 Decision C , DRDR decision-makers determine which 
proposals will be funded (or committed to ) and those that will be 
rej acted, 

3,7 Rej act , DRDR decision-makers, assisted by DRDR 
evaluators. Identify the unsuccessful proposals and notify the 
interested groups or institutions of their rejection, 

3,9 Recycle . DRDR decision-makers, assisted by DRDR 




evaluators, identify proposals which potentially are acceptable 
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and notify the Interested groups or institutions of what modifications 
must be made. 

4.0 Make comm I tme n t * DRDR makes commitments to successful 
groups or Institutions* 
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1 1 1 

Illustrations of the General System Model 

This chapter illustrates the evaluation model described 
in chapter two. Specifically, this chapter provides walk throughs 
for decisions 2, 4, 6, and 9* These particular decision points 
are illustrated because they represent the range of evaluation problems 
implied by the support policy. In addition, they encompass many If not 
all of the most important decisions serviced by the proposed evaluation system. 

Application to Decision 2 - New Institution Review 

Decision 2 - Should a given proposal to start a new institution be 
funded? 

DRDR, after identifying funds that can be utilized, decides to start 
new r & d institutions. This announcement is circulated, proposals 
are received and reviewed, and the three - six month planning grants 
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are awarded to app rox i mate 1 y three times as many groups as will be 
funded. For illustrative purposes, assume nine planning grants are 
competing for three eventual contracts, St is anticipated that all 
three of the successful groups will grow to become mature institutions. 

Guidelines (see appendix A) are sent (1.0) to each of the 
nine groups working on planning grants and proposals are received and 
reviewed (1.1) by the specified date. A contractor is selected to 
perform analyses according to specifications. In the meantime, 
evaluation monitors determine if guidelines have been followed and 
if all required information is included in the proposal (1.2), This 
task requires about three man-days. If a proposal is inadequate, 
it is recycled for revision (1*3) and one week is allowed for its 
return. Then DRDR evaluators attempt to verify information contained 
in the proposals (2.2). They note information regarding such things 
as supporting i nst i tut ional commitments (use of data processing 
facilities, etc.), personnel commitments, and research findings. 

They verify the accuracy through phone calls to knowledgable people. 
About one-half day is spent on each proposal. Any information 
remaining in question is noted for later referral to the site 
visitors. 

Expert judges have been identified according to systematic 
procedures described in "Selection, Training, and Assignment of 
Judges," From this pool, nine are identified (2.0) who have expertise 
in judging management, planning, and personnel development* By 
specified procedure, DRDR assigns the nine judges to read the nine 
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proposals (2.1) and they are circulated so that each is read by 
th ree j udges * 

After the DRDR evaluators complete their verification checks 

(2.3) , the proposals, along with all instructions. Instruments, and 
materials required, are sent to the judges for review and assessment 

(2.4) . 

One week is allowed for assessments to be performed. Judges 
are allowed one man-day per proposal to review, to write the report, 
and to respond to the instrument. Three independent reviews are made 
of each proposal; consequently, twenty-seven man-days are required 
of the nine judges. One week Is scheduled for this review. 

While reviews are being conducted, DRDR evaluators make 
plans for the analysis to be conducted. As soon as readers return 
documents (2.5), DRDR evaluators collect and organize assessment 
instruments. These evaluators then send instruments (2.6) to an 
analysis contractor where they are analyzed and rank orders are 
obtained. Two weeks are required. Monitors, in the meantime, 
prepare for meeting with DRDR decision-makers. 

After data are returned DRDR evaluators review the analysis 
(2.7) and prepare for internal review (2.8). DRDR decision-makers 
consider the strengths and weaknesses of each section of the nine 
proposals (2.9), and divide the proposals into the top three and 
bottom six (2.10). The six unsuccessful applicants are notified 
(2.11) and the other three are told of the acceptance of their 
proposa 1 (4. 0) . 
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Application to Decision 4 - Maturity Review 

Decision 4 - Is the capability of the ? nst I t u t i on to manage its 
basic program p 1 an s a t i s f ac to r y ? 

Several new institutions (for this i 1 1 us t ra t Ion , assume three) 
received planning grants of six months, moved from planning into 
the new^ institution phase, and after a period of two years passed 
successfully into the deve 1 op i ng^ i ns 1 I tut ion phase. During the 
successive two - four years these three developing institutions 
worked on a program of r & d activities, developed their staffs and 
planning and management capabilities, and arrived at a most 
significant point in their history - a review to determine if they 
have the institutional capabilities to manage a program plan and 
thus be designated as mature r £■ d institutions. This Is the last 
Lime DRPR, having nurtured them from infants to maturity, examine 
carefully their institutional capabilities. 

Guidelines (see appendix A) are sent (1.0) to the three 
Institutions five months before their fiscal year ends, with 
instructions to respond by a deadline date set at least three months 
before the end of the fiscal year (to allow two months for the 
decision and one month for negotiating). The institutions also are 
told to expect site reviews to begin about forty-five days after their 
reports are received in DRDR. The site reviews will involve five 
DRDR- selected expert judges. Their names will be made known to the 
institutions at least twenty-five days before their visit. The 
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institutions also will be informed of the questions and criteria that 
site visitors will have and will be given a chance to review and 
respond to the reviewers 1 reports. 

On the specified date, the three reports (responses to 
guidelines) arrive at DRDR. The DRDR evaluator checks the three 
reports for content (1.1), He determines if all information called 
for in the guidelines Is in fact included. If any required content 
is missing* he returns the report, notes the needed additional 
information* and allows one week for its return; otherwise, reports 
are forwarded for further analysis (1.2). Reports that contain the 
required information are examined by DRDR evaluators (2.2) . Any 
questionable items are noted and held for reference to the site 
v I s ( tors (2,3). 



procedure that covers the areas of expertise needed (2,0), The 
emphasis is on judging planning, management, and personnel of the 
i nst i tut i on . 



criteria, dates, etc.) about the site review is sent to the interested 
institutions. Each judge assigned the three reports is given an 
instrument for recording his assessments (2,4). 



When the judges return the reports (2.5) the DRDR evaluator 
collects and organizes the assessment instruments and sends them to 



Five expert judges are selected according to the specified 



DRDR then assigns the judges to read the reports and assess 



them according to specified criteria (2,1), Information (names. 



the contractor for analysis (2,6). Two weeks are allowed to perform 
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the analyses. After analyses are returned; DRDR evaluators review 
them (2.7) and summarize the ranks for the DRDR dec i s i on-make rs 
who review each ranked section (2.8) and determine their position 
(2.9). At least three of the five expert judges should recommend 
"passing", Rank order of the three reports by sections reveals 
their strengths and weaknesses (2.10). Therefore, any one of the 
i nst I tut iona 1 reviews may be concluded (2.11) at this point with the 
determination to ph ase out, support , or to recycle. DRDR notifies 
each institution of its decision. 

For purposes of the illustration, assume a site visit is 
made to each of the three (3*0). The evaluator notifies the judges 
of the site-visit schedule. Each judge is assigned to visit the 
institution whose report he reviewed (3-1). Each visit is conducted 
during a two-day period and one day is allowed each judge for 
compiling his report (3*2), Thus, forty-five man-days of judges' 
time are required to conduct the visit and compile the reports. 

The visits are scheduled to allow two weeks for visiting and filing 
the reports. Visits are conducted according to the plan outlined 
in "Procedures for Site Reviews" (see chapter five). 

Assessments then are received at DRDR (3-3) and the 
instruments are sent to the contractor for analysis (3-4). One week 
is allocated for analysis. 

After receiving the analyses from the contractor (3,5) the 
DRDR evaluator prepares for the Internal review (3.6). DRDR decision- 
makers then consider the relative strengths and weaknesses of each 
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institution by areas of concern, examine the rank-order data as well 
as overall recommendations, and make decisions about each institution 
(3-7)- The division has three alternatives: phase out support (3*B), 
recycle one more year (3*9), or award status as a mature institution 
(4,0), Data should enable a defensible decision to be made - given the 
constraints, priorities, and policies of DRDR, DRDR then implements decisions. 

Application to Decision 6 - Program Plan Review 

Decision 6 - Should the Office of Education commit itself to support 
qiyen proposed new program plans ? 

Mature institutions have been invited to submit proposals to initiate 
new programs for training technologically skilled but unemployed 
persons to become env ? ronmenta 1 -cont ro 1 specialists who can help 
n clean up the env i ronment , 11 Five proposals for five-year grants 
approximating $3,500,000 are received. The funding of one or more 
of these represents a substantial portion of the DRDR budget; and 
such a program would have high social significance. Thus, the 
five proposals require thorough evaluation. 

Assuming that decision A is positive, DRDR identifies 
expert judges in a stratified classification (2.0) and randomly 
assigns five of them to read the five proposals (2,1). Meanwhile, 
others at DRDR will verify the authenticity of the information in the 
documents presented by the institutions (2.2) and later refer 
discrepancies to site v i s i tors (2 . 3) , Once the judges receive copies 
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of proposals, guidelines, and evaluation forms (2*4), they 
independently submit their judgments on each of the criterion measures 
or dimensions of the evaluation instruments. Considerable attention 
Is given to determine if the proposals furnished positive evidence 
of meeting such criteria as (1) fulfillment of DRDR priorities and 
of long- and short-term needs of identifiable social groups In the 
nation, (2) sound program planning for successful operation including 
built-in evaluation procedures, and (3) designation of milestone 
reviews for attaining specific objectives. After the data 
corresponding to the judgments of the five expert evaluators 
are received (2,5) they are sent to an external agency for 
analysis (2,6), returned (2,7) and then summarized in terms of how 
the proposals ranked on each of several relevant criterion 
dimensions (2.8), Painstaking internal review at DRDR 
subsequently takes place. Not only the rankings, but also 
the availability of funds and the existence of competing priorities 
and prior commitments, are carefully weighed and interpreted (2*9), 
Decision B (2,10) is then made to select the top-ranked proposal, 
at least tentatively. The four unsuccessful proposers are notified 
( 2 . 11 ). 

In view of the magnitude of the potential commitment, the 
third stage of the System Model involves observing the capabilities 
of the institution for implementing the program plan, A site 
visitation Is required (3.0), The same five expert judges who read 
the proposals automatically become site visitors (3-1) and 
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proceed to the Institution two or three weeks later (3-2). The site 
visitors not only try to reconcile any discrepancies referred to 
them (2.3), but they also examine first-hand the capabilities of the 
Institution, Once the data on the evaluation Instruments are received 
from the judges (3-3), they are sent to the contractor for data 
analysis (3-4) and returned a few days later (3-5)- These data are 
subjected to careful internal review and are checked against 
information presented in the proposal. Once again, competing 
priorities, budgetary resources, and other commitments are studied 
(3-6), Finally, on the basis of the ranking assigned (3-7)s 
decision C is made either to reject or to accept the proposed 
program. Acceptance of the proposal means that a commitment (4.0) 
is made for funding a proposed new program plan. 

Application to Decision 9 - Program Review 

Decision 9 - Di d the program reach its objectives? 

Assume that a mature Institution with a four-year grant of $1,200,000 
to prepare instructional materials emphasizing an inductive, problem- 
solving approach In high school physics has reached the terminal- 
review stage. At this point a report is filed with DRDR so that it 
can decide how adequately the program attained the objectives and 
goals that it set for itself and how well the product represented 
by elements in the final report meets acceptable standards. Also, 
the terminal review (without a site review) allows DRDR to ascertain 
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if the products are harmonious with societal needs and priorities 
and with expectations of current educational audiences and if they 
are competitive with similar products from other agencies. 

Three expert judges are identified and assigned (2*0 and 
2.1) to look at the final report assigned to them (2,4) and to make 
assessments. The judges use certain criteria pertaining to the 
importance, quality, and probable usefulness of elements cited in 
the final report and examine samples of instructional materials 
like teachers 1 manuals, workbooks, textbooks, film strips, testing 
exercises, and examples of students* work. In assessing the quality, 
the expert judges rate the design, accuracy, and comprehensiveness 
of Instructional materials plus their suitability, cost and 
marketability for the target population. Any supporting evidence 
such as sales figures, journal reviews, testimonial letters, 
minutes of school-board meetings, and Information from phone calls 
to teachers and administrators is considered. 

Once the judgments are received and recorded as data (2.5), 
the information Is sent to an external agency for analysis (2,6), 
After it is returned. Its internal analysis (2.7) occurs, followed 
by a summary analysis involving a scale value (ranking) (2*8), An 
internal review by DRDR decision-makers (2.9), constituting an 
I nterpretat I on of the analyzed data in relation to the product and 
its elements as well as the degree to which certain criteria were 
met, leads to decision B (2.10) regarding how satisfactory the 
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If marked discrepancies are oted between what was stated 
in the final report and the actual state of affairs, steps 3.0 - 3-8 
involving sequential activities of a site visitation and its 
accompanying evaluative judgments follow. Such action leading to 
decision C is unlikely in a mature institution. 
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Decisions, Criteria, and Guidelines 



So far this report has responded to the charge by presenting a general 
evaluation model and by Illustrating how that model applies to decisions 
associated with the new DRPR support policy. DRDR also specified that 
the task forces suggest criteria to be applied In the evaluation 
associated with each of the ten decisions. 

Consequently, this chapter presents a master list of 
criteria believed to be sufficient for servicing the ten decisions. 

The ch' ter also analyzes how the criteria apply differentially to 
the ten decisions. 

The ten decisions require the use of four relatively distinct 
sets of criteria. Decisions 1 and 2 are served by criteria associated 
with the prospects for developing a new j ns t i tut I on ; decision 4 is 
served by criteria concerned with the quality of _past effort s to develop 
new j ns t i tut Ions ; decisions 3* 5* and 6 are served by criteria for 
judging the worth of a proposed p rog ram; decisions 7* 8, and 9 are 
served by criteria for judging completed program activities; and 
decision 10 is served by al 1 criteria. 
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Analysis of How Criteria Relate to the Ten Decisions 



Each of the ten evaluation types implies certain criteria appropriate 
to the decisions to be made at that point* The actual criteria listed 
were derived as follows* each member of the Advocate T earn , after 
thorough study of a series of documents provided by DRDR, after a 
day-long briefing related chiefly to the problems attendant on 
current evaluation practices, and after considerable discussion by 
the team, prepared a listing of criterion nominations differentiated 
in relation to the ten evaluation types. The complete listing of 
the 357 nominations derived from this process was then put through 
several iterations of a classification process. 

It became clear from the manipulations of this process that 
there were two major dimensions along which the criteria needed to be 
cl ass i f i ed, 

1 nst I tut ion-program dimension . Certain criteria related 
primarily to institutional characteristics, while others related 
primarily to program characteristics. 

Time dimension (d i chotomousl y characterized as a prospective - 
retrospective dimension ). Certain criteria were forward looking and 
called for proposals; others were backward looking and called for reports. 

This bilateral dimensionality suggests that the criteria can 
most conveniently be organized in the form of a 2 x 2 table such as 
shown in table 1. Certain criteria were appropriate to prospective 
I n s 1 1 1 ut i on $ , others to p rospect i ve p ro grams , others to retrospect i ve 
instl tut ions , and still others to retrospective programs . The cells 
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relating to these four classifications are labeled, for ease In 

reference 1, 2, 3, and 4 respect i vel y. Criteria in cells 1 and 2 

are applied to proposals; criteria in cells 3 and 4 are applied to reports* 



Table 1 

Basic Criterion Format 





Organ i zat i onal Level 


1 nst i tut i on 


Program 


Vi ewpoi nt 


Prospective 

(Proposals) 


i 


2 


Ret rospect 1 ve 
(Reports) 


3 


b 



Each of the ten evaluation types may be classified as calling 
for a prospective institution evaluation, a retrospective institution 
evaluation, □ prospective program evaluation, or a retrospective program 
©valuation. Figure 2 displays the classifications made by the advocate 
team. Nine of the ten evaluation types fall conveniently Into one 
of the four criterion classification cells. Decision 10 ? Budget 
and Contract Review, contains elements of all four. 

As will be seen, the criteria found in these four cells show 
a certain parallelism, but they are by no means identical. Generally 
the criteria can be contained adequately in terms of ten major 
classifications,^ but not all classifications apply equally either in 




^Thcse classifications are incomplete as of this writing since 
not all 357 originally nominated I terns have been successfully assigned. 
Minor variations (chiefly additions) may occur at a later date. 
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Figure 2 



Types of Evaluation Required for Each Decision Point 





!#= 

1 nst I tut ion 


Both 


Program 


Prospective 


1 . 1 nst I tut i onal 
PI anni ng-Grant 
Awa rd 

2. Institutional 
Ope rat I on- Grant 
Award 




3 . Program-Plan 
App rova 1 

5* Program Planning- 
Grant Award 

6, Program Plans 
App roval 


1 

Both 




10. Annual Budget 
and Contract 
Rev I ew 




Retrospective 


4. Managements 
Capab I 1 I t i es 
Cert i f I cation 




7 * Milestone Review 

8 e Unanticipated 
Modi f i cation 
Review 

9. Program Outcome 
Eval uat i on 
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whole or In part to the four cells. To clarify, the criteria are 
displayed In table 2 . The complete criteria list Is found on the 
vertical dimension with the four cl as s I f I cat I ons for criteria found 
across the horizontal dimension. Y 1 s (for yes) and N's (for no) in 
the cells indicate which classes of criteria apply to each criterion 
in the master list. Superscripts over the Y’s and N's denote special 
meanings as explained in the footnotes. Since it is known that the 
different classifications of criteria apply d i f fe rent I al 1 y to the ten 
decisions, table 2 is useful in selecting criteria and developing 
instrumentation to service the different decision situations. 

The criteria are embedded in two sets of documents appearing 
in the appendix A: 

-- ’’Guidelines for applicants 11 for DRDR support that Instruct 
the applicants to supply information needed by the evaluators to apply 
all the criteria relevant to each pending decision, and 

-- “Guidelines for judges 11 to direct applicants 1 attention 
to key Information In the applications and to supply the criteria for 
judging then.. 

The two categories of guidelines are Interlocked at every point 
so that applicants are directed to provide what the judges need; conversely, 
the judges are directed to inspect what the applicants provide. The 
criteria for judgment are the points In common between the two sets of 
guidelines; i . e. , both the applicants and the judges are told the basis 
on which DRDR wants to rest its decisions. 

Full sets of guidelines appear In the appendix A for decision 
points 1 - 6, For decision points 7 - 10, where the DRDR evaluation staff 
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Master List of Criteria 



Table 2 



Master List of Criteria Cross Referenced 
to the Ten Support Policy Decisions 



Types of 
Cri ter I a 


P rospect i ve 
I nst I tut I on 


P rospect i ve 
Program 


Retrospect i ve 
Inst I tut ion 


• Retrospective 
Program 


Dec! slons 
Served 


1. 2, 10 


3, 5, 6, 10 


4, 10 


1, 8, 9, 10 


1 # Significance c c focus 


Y a 


Y b 


Y a 


yb 


A* Priority 


Y 


Y 


Y 


Y 


1. Congruence with 
government 
priori ties 


Y 


Y 


Y 


Y 


2, Congruence with 
accepted values 


Y 


Y c 


Y 


Y 


3, Congruence with 
nat i onal 
professional 
educat i on 
priori ties 


Y 


Y c 


Y 


Y 


4, Congruence with 
national social 
and economic 
p r i or i t i es 


Y 


yC 


Y 


Y 


5* Relationship to 
pupil, 

demographi c, 
and economic 

targets 


Y 


Y c 


Y 


Y 


6, 1 nt r i ns i c 

s i gn i f i canoe 


Y 


Y 


Y 


Y 


; B, Amenability to an 

r & d approach 


Y 


Y 


Y 


N 


C, Durab i 1? ty and 
reel di v? sm 


N 


N 


Y 


Y 


D # Programmatic 
natu re- -cap a b 1 e 
of programmatic 
exploration 


Y 


N 


Y 


N 



a = Significance of focus (Mission) 
b = Significance of focus (Program Goal) 

c - These items should be used only with proposals coming from mature 
institutions that have not undergone evaluations 1 - 4, 
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Types of 
Cr i teri a 


Prospect i ve 
I nstl tut i on 


Prospect I ve 
Program 


Retrospective 

Institution 


Retrospective 

Program 


Dec! s I ons 
Served 


1 . 2 , 10 


3, 5, 6, 10 


4, 10 


7 , 8 , 9 , 10 


II. Significance of 
outcomes 


Y d 


Y* 


Y® 


Y e 


A, Direct benefits 


N 


Y 


Y 


Y 


B, Indirect benefits 


N 


N 


Y 


Y 


C, Negative side 
effects 


N 


N 


Y 


Y 


D. Pervasiveness 


N 


N 


Y 


Y 


E* Tfmel i ness- 
cri 1 1 cal 1 ty 


N 


Y 


Y 


Y 


F, Exportab ? 1 i ty 


N 


N 


N 


Y 


1 1 1 , Uni queness 


Y 


Y 


Y 


Y 


A, Substantive 
uni queness — 
onl y agent 
exploring area 


Y 


Y 


Y 


Y 


B* Geographic 
uni queness-- 
dfstinct regional 
contribution 


Y 


Y 


Y 


Y 


IV, Viability 


Y 


Y 


Y 


Y 


A. Management 
capab i 1 i ty 


N 


N 


Y 


N 


B. Planning 
capabi 1 i ty 


Y 


Y 


N 


N 


C, Fiscal 

respons i b i 1 i ty 


N 


N 


Y 


N 


], Fiscal 
control s 


N 


N 


Y 


N 


2, Program 
budgeting 


N 


N 


Y 


N 
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d = Significance of prospective outcomes 
e - Significance of achieved outcomes 

1+2 
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Types of 
Cr f ter i a 


P rospect T ve 
1 nst I tut i on 


Prospect I ve 
Program 


Retrospective 
I nst i tut Ion 


Retrospective 

Program 


Dec! s ! ons 
Served 


1 * 

K> 

w 

o 


3, 5, 6 , 10 


4, 10 


7, 8, 9, 10 


3. Adequate 
report i ng 


N 


N 


Y 


N 


D. Political, legal, 
soci al , and mora 1 
vi ah 1 1 i cy 


Y 


Y 


Y 


N 


E. Cooperab i 1 I ty-- 
coopt ab i 1 i ty 


Y 


Y 


Y 


Y 


I , Endorsement 
and assistance 
from other 
i n s t i tut i ons 


Y 


N 


Y f 


N 


2. Potential for 
cooperat i on 
and 

col 1 aborat i on 


Y 


Y 9 


Y h 


Y 9 


3- Potential 
for 

en I I stment 


Y 


N 


yJ 


N 


4* Ava liability 
of needed 
servf ces 


N 


Y 


YJ 


N 


5 * Relation to 
ea rl f er work 
of other 
I nst i tut i ons 


Y 


Y 


N 


Y k 


6, Existence of 
commun i cat i ons 
network 


N 


N 


Y 


N 


F. Parity 


Y 


Y 


Y 


N 


!. Representation 
from cl I ent 
groups 


Y 


Y 


Y 


N 


2. Participation 

from key 
aud i ences 


Y 


Y 


Y 


N 


3p Representation 
from various 
di scipl ines 


N 


N 


Y 


N 



f - Assistance from other institutions 

9 ™ Potential for cooperation and collaboration with other programs 
h = Cooperation and collaboration 
i = Enlistment 43 

j = Adequacy of needed services 
k = Relations to earlier work of institution 



Types of 
Criteria 


Prospective 
I ns t i tut 1 on 


Prospective 
P rogram 


Retrospective 
I nst i tut 1 on 


Ret respect 1 ve 
Program 


Dee! s 1 ons 
Served 


1, 2, 10 


3, 5, 6, 10 


4, 10 


7, 8, 9, 10 


4, Rep resentat 1 on 


N 


N 


Y 


N 


from various 
techn I ca 1 
fields 










G, Practi cal I ty 


N 


Y 


Y 


N 


1 - Scope of work 


N 


Y 


Y 


N 


2. Avat 1 ab 11 I ty 


N 


Y 


Y 


N 


3- Schedule 


N 


Y 


Y 


N 


H. Geography 


Y 


N 


N 


N 


1 * Access i b 1 1 1 ty 


Y 


N 


N 


N 


2, Ability to 


Y 


N 


N 


N 


recru i t 
personnel 










3, Ability to 


Y 


N 


N 


N 


cooperate with 
other 

1 n st i tut i ons 










1, Personnel 


Y 


Y 


Y 


N 


1 , Competence 


Y 


Y 


Y 


N 


2* Ava liability 


Y 


Y 


Y 


N 


3. Interest 


Y 


Y 


Y 


N 


J. Faci 1 i t i es 


N 


N 


Y 


N 


1 , Space 


N 


N 


Y 


N 


2, Equipment 


N 


N 


Y 


N 


V. Planning and 


Y 


N 


Y 


N 


evaluation adequacy 










A, Means to assess 


Y 


N 


Y 


N 


and modify goals 








B. Means to identify 


Y 


N 


Y 


N 


and assess 

al ternat ? ve 
strategies 
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Types of 
Cr ! ter i a 


Prospective 
1 nst i tut ion 


Prospective 

Program 


Ret respect 1 ve 
Institution 


Retrospective 

Program 


Dec! slons 
Served 


1, 2, 10 


3, 5, 6, 10 


4, 10 


7, 8, 9, 10 


C. Means to 
monitor and 
assess 
operational 
act ? vl t i es 


N 


N 


Y 


N 


D« Means to 
daterrn i ne 
achievement of 
ob j eet ! ves 


N 


N 


Y 


N 


E, Means to 

relate budget to 
p rograms 


N 


N 


Y 


N 


VI. Program balance 


N 


N 


Y 


N 


A* Necessary and 
sufficient 
program set 


N 


N 


Y 


N 


1. Cumulativeness 
--log! cal 
relation of 
goals at 
every level 


N 


N 


Y 


N 


2. Scope“ 

suf f I cl ent 
spectrum of 
p rogram 
activity to 
cover needs 


N 


N 


Y 


N 


B . Adequate balance 
of resource 
al locations to 
P rograms 


N 


N 


Y 


N 


C. Adequate balance 
between r & d 
act I v! ties 


N 


N 


Y 


N 
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Types of 
Crt ter i a 


Prospect I ve 
! nst i tut Ion 


Prospect 1 ve 
Program 


Retrospective 
1 nst i tut ton 


Retrospective 

Program 












Deci s i ons 
Served 


1, 2, 10 


3, 5, 6, 10 


u, 10 


7, 8, 9, ,10 


VII. Adequacy 

program elements 


N 


Y 1 


N 


Y 


A. Objectives 
sped f I ed 


N 


Y 


N 


Y 


B. Relevance of 
objectives to 
program and 
i nst l tut fonal 
goal s 


N 


ym 


N 


Y 


C. Adequacy of 
p rocedu ral 
des i gn 


N 


Y 


N 


Y 


D* Met hodol ogf cal 
adequacy 


N 


Y 


N 


Y 


E, App rop ri ateness 
of schedule 


N 


Y 


N 


N 


F # App rop rl ateness 
of personnel 


N 


Y 


N 


N 


G, Adequacy of 
fad I T t Fes 


N 


Y 


N 


N 


VIM, Cost 


V 


Y 


Y 


Y 


A. Budget 


Y 


Y 


Y 


Y 


1 , SuF tab fifty 


Y 


Y 


Y 


Y 


2, Sufficiency 


Y 


Y 


Y 


Y 


B. Time 


Y 


Y 


Y 


Y 


C, Personnel 


Y 


Y 


Y 


Y 


IX. Economic 

ef f i ci ency 


N 




Y 

1 


Y 



1 — Adequacy of prospective program plans 

m = (Not relevant for general-purpose mature institutions such as 
AIR or ETS) 
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Types of 
Cr I teri a 


Prospect I ve 
Inst i tut I on 


Prospective 

Program 


Retrospective 

Institution 


Retrospective 

Program 


* 


Dec! % i ons 
Served 


1. 2, 10 


3, 5, 6, 10 


4, 10 


7. 8, 9, 10 




A, Unit costs 


N 


N 


Y 


Y 




B. Cost/benefft 
rat I o 


N 


N 


Y 


Y 




X. Potential for 
outside support 


Y 


N 


Y n 


N 





n - Some outside support achieved 
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exercises less responsibility for gathering and assessing « nformat i on, 
responsibilities are described but guidelines are not supplied for 
reasons given In the appendix. 

Guidelines appearing In appendix A do not conform to the 
classifications of criteria that have been presented in this chapter. 

The guidelines In the appendix were prepared simultaneously with the 
development of chapter four; it was not apparent until late in the 
work of the Advocate Team that the criteria could be parsimoniously 
classified and applied to decision situations according to the 
categories of retrospective and prospective programs and institutions. 

Neve rt he less, the guidelines In the appendix will prove useful to DRDR 
In launching its evaluation system. However, revisions of the guidelines 
should begin immediately to rearrange them to conform with the cl ass i f i cat i on 
system that appears in this chapter. With added time* the Advocate 
Team or another outside agency could perform this task for DRDR, 
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V 

Special Procedures 

The combination of the new support policy and the model for evaluation 
proposed In this report calls for some departure from existing evaluation 
procedures. This chapter makes specific suggestions for modifying DRDR 
evaluation practice. The first part of the chapter provides recommenda- 
tions for the selection, training, and assignment of expert judges. If 
DRDR follows these recommendations, credibility, reliability, and 

objectivity are insured for the proposed evaluation system. 

The second part proposes instrumentat ion and quantitative 
procedures to be used in obtaining and analyzing comparative judgments. 
This part discusses the all-important control mechanism as embodied in 
the expert judges. They are expected to make independent judgments 
that later can be submitted to psychometric analysis to determine their 
reliability and the rank order of the objects of the judgments. 

The third part suggests explicit procedures for site reviews to 
standardize the conditions under which judgmental data are gathered. 
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These procedures will Insure that assessments are obtained from each 
judge Independently, 

The final part outlines procedures to be followed in conducting 
audits of the new evaluation system. 



Expert Judges 



The Advocate Team recommends that DRDR Ins ta 1 1 the sys temat i c procedu re 
deserf bed below for the pu rpose of select! ng , t ra i n ? ng , and assigning 
expert judges to read and rate p roposa 1 s and reports , 

To improve the reliability and validity of evaluating 
proposals at various decision points, certain procedures for the 
selection, training, and assignment of readers are desirable. Although 
the U. S. Office of Education maintains a large file of external 
reviewers that can be forwarded to DRDR, it is necessary to update 
this file and to develop pools or subpopulations of readers who 
will correspond In terms of their competencies to certain broad 
substantive areas In education. 

Once subpopulations are identified in relation to broad 
areas of classifications and In relation to the need for them to 
serve at designated decision points in the Maturity Model, a random 
sample of readers will be chosen and asked to evaluate proposals with 
the understanding that they also will be expected to participate as 
site vi s ! tors # 

After these individuals are identified, they will come to 
Washington to participate in a two- or three-day training session. They 
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will practice carrying out evaluations with two or three rep resentat i ve 
proposals* Such a session will help to achieve a common frame of 
reference that will increase substantially the reliability of the evalua- 
tion process by establishing a comparable level of understanding of 
the directions in documents and the procedures to be followed; defining 
certain terms unique to the DRDR operation; and familiarizing participants 
with the nature of the rating process in evaluation of programs and 
institutions so that common, systematic errors in response style can be 
minimized. After such training, tho readers will be given a common 
experience with a new proposal to check on the reliability of judgments 
and to identify potential evaluators who might be divergent in their 
responses. If the divergence appears to be due to the inability of 
the evaluator to participate effectively in the mechanics of the 
rating process rather than to a difference in philosophic stance, he 
can be given additional training or dropped. (The estimates of 
reliability could be obtained from standard techniques of analysis 
of variance or co rre 1 at Iona 1 methodology.) 

The follow-up activity of evaluating a triai proposal that 
could serve as an anchor or linkage mechanism to evaluations of other 
proposals serves still another purpose. It affords an approximate basis 
for DRDR to make some allowance for subsequent differences among the 
ratings of evaluators that would still exist after they had received 
supervised training. The determination of normative data for the 
responses of expert judges rests on two assumptions, an evaluative 
instrument covering important criterion dimensions has been developed for 
use by evaluators and different forms of this instrument contain 
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several common I terns or anchor dimensions for equating purposes. This 
point is explained in the next section. 

From each of the pools of potential DRDR evaluators in a given 
substantive area a random assignment of three to five readers - depending 
on the decision point involved - will be made to each proposal. (When 
a decision point involves a substantial investment of DRDR resources, five 
evaluators will be assigned; at a point of relatively less importance 
a committee of three readers will suffice.) Spuriously high estimates 
of reliability, resulting from the tendency of some evaluators to be 
swayed toward the opinions of dominate or prestigious persons, will be 
avoided by requiring each individual to evaluate independently and 
privately. Quality control of the evaluation process and the continued 
realization of maintaining its reliability at a subsequent point in the 
Maturity Model also will be enhanced by having the external evaluators 
participate in onsite visits at institutions that contributed the 
proposals they reviewed. 

Instrumentation and Quantitative Procedures 

The Advocate Team recommends that DRDR employ the i nstrumentat ion and 
quant i tat ive procedures described below for maki ng comparative j udqments . 

A methodology must be developed to rank accurately the submitted 
proposals and reports. Thus DRDR should prepare a comprehensive 
evaluation instrument reflecting the criteria developed by the Advocate 
Team adapted to meet the requirements of different decision-points in 
the Maturity Model and devise appropriate quantitative procedures to 
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determine comparative rankings of expert judges for proposals and reports 
within given broad substantive areas and to the extent possible, across 
different substantive areas. Furthermore, a contractual agreement should 
be made with an external agency (e.g., Educational Testing Service) to 
develop procedures for analyzing data from expert judges for arriving 
at reliable comparative rankings; to apply these procedures to the 
analysis of data derived from the different forms of the instrument that 
DRDR has devised for evaluation; and to communicate these resulting 
rankings to DRDR. This agency also can help identify potential expert 
judges and develop procedures for assigning judges, to rating tasks and 
site visitations. 

Development and scoring of evaluation instrument . The 
following steps guide the preparation of an evaluation instrument and 
the development, use, and monitoring of a scoring system adapted to meet 
the requirements of different decision-points in the Maturity Model: 

“ A scale will be constructed, corresponding' to each of the 
major criterion variables previously identified at each decision-point 
(see chapter four), and incorporated within the two sets of guidelines 
(see appendix A). Its items or questions will represent an operational 
definition of subcriteria or objectives subsumed under each major criterion 
dimension. 

-- Each form will contain certain criterion dimensions and thus 
questions or items common to other forms. Consequently, a core of anchor 
items ensues that can be used as linkages for an approximate equating of 
judgments at different decision-points rendered by different expert judges 
or for studying gross changes in judgments from one decision-point to 
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another. (The inclusion of a certain number of common anchor items in 
all forms of the evaluation instrument also wi 1 ! permit a rough equating 
of differences in the recorded perceptions of various judges in somewhat 
different content areas at varying points of decision in the Maturity Model.) 

-- A scoring system will need to be derived for the questions 
or items within each criterion dimension so that points can be accumulated 
within each dimension to afford a basis for comparative evaluations 
across expert judges. 

— Once appropriate s tanda rd i zat i on procedures are devised to 
permit the conversion of data from different scales to a common base 
(standard scores having the same means and same standard deviations), 
a method will be afforded at each decision-point for drawing profiles 
that indicate on each of the proposal or program as evaluated by each 
judge. Although subject to a considerable error of measurement as a function 
of the few questions within each criterion dimension, such profile will 
furnish (1) a gross, diagnostic picture of strong and wea' oints that 
will have important feedback value to the institution or roposal writer; 

(2) a means of examining systematic differences in the e\uluations of the 
expert judges at a given decision-point or from one decision-point to another; 

(3) a basis for studying differences in the perceptions of expert judges 
on each proposal and across proposals at a particular deci s i on-poi n t ; 
and thus (4) a vehicle for monitoring the responses of expert judges. 

-- Once a weighing system has been derived for each of the 
criterion dimensions at a particular decision-point in the Maturity Model, 
the comparative standings of the proposals will be added over the 
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several dimensions so that a final number Is obtained for each proposal 
to permit its being given an overall rank value*. 

-- Whenever appropriate, estimates of the reliability of 
scores at each criterion dimension and for composite scores over 
several dimensions will be obtained. Failure of an individual judge s 
assessments to correlate with those of other judges will suggest an 
examination of his form to check for any obvious errors in the recording 
process or id iosyncrac i es in his response style. Depending on the 
circumstances surrounding the evaluation, it may be necessary to obtain 
the judgments of an additional judge. Such a necessity, however, will 
be avoided when possible# 

Alternative approaches to determining comparative jud gments. 
Several alternative quantitative procedures must be explored in arriving 
at comparative judgments of the quality of proposals and reports, both 
within a given content area and across different substantive areas. 

DRDR anticipates that two rather distinct situations may arise in the 
number of proposals to be submitted: the submission of approx imatel y 

five to ten proposals in a given substantive area--the more likely 
occurrence, and the submission of as many as forty or fifty proposals in 
a particularly popular area of interest. 

Although somewhat different adaptations of methodologies 
probably will be required in each of these two situations, the basic 
methodology will be similar. The external agency previously cited 
will be requested to explore several alternative quantitative procedures 
to allow the establishment of reliable and valid comparative judgments 
for each of these two situations,. 




62 , 



56 



Tv/o methodologies to which this agency might give serious 
consideration are triwSe of pa i red comparisons and item sampling with 
which psychometricians are familiar. The paired comparison technique 
involves the taking of all possible permutations of pairs of objects 
(e.g., proposals) in a given set and judging which one of the objects 
in the pair is of higher value or holds greater appeal. A relatively 
complex arithmetical procedure leads to the determination of scaled 
Items. In the instance of an evaluation instrument that does not 
contain enough items or questions in a criterion dimension for reliable 
scoring, the paired comparison approach is a promising one for 
obtaining scale values if the number of objects (proposals) does not 
exceed ten or twelve. 

For the circumstance of forty or fifty proposals, it seems 
reasonable to believe that if the overlapping of proposals in samplings 
of ten or twelve proposals can be introduced to the extent of three 
or four proposals, a mathematical procedure can be developed for equating 
the scale values from the paired comparisons in one sampling of proposals 
to another sampling. Thus a means will be afforded for arriving at 
comparable scale values for the entire group of forty or fifty or more 
proposals. This procedure in essence combines features of paired com- 
parisons and item sampling methodology. Comparative judgments might be 
obtained only on single dimensions or on clusters of highly correlated 
dimensions. A working methodology to permit the determination of scale 
values of many proposals will constitute an important breakthrough In the 
comparative evaluation work of the DRDR. 
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Procedures for Sire Reviews 

The Advocate Team re contends that the procedures described below be 
adopted to govern all DRPP^sponso red site reviews and that the ensuing 
reports of these rev j ev/s be utilized by DRDR In its dec f s lon-makl ng processes . 
Site reviews are an important element for collecting and 
verifying information needed to make decisions regarding the 
establishment of r £• d institutions and the funding of their programs. 

Of course, the basic information required for DRDR decision-making 
will be submitted by the institutions in response to DRDR requests* 

Thus the purposes of site reviews are to verify the accuracy of the 
information submitted, to confirm or correct the interpretations 
placed on that information, and to improve the information base for 
DRDR decisions* 

Consequently the following procedures are outlined: 

1. Government officials will select site visitors (except 
for decision 7 where the r & d institutions have the authority for 
designing milestone evaluations and choosing any site visitors who 
will be involved). However, the names of site visitors will be 
submitted to the institutions for them to challenge. The final 
decision on the choice of site visitors, nevertheless, (excepting 
decision 7) remains with the government officials. 

2. Guidelines used by site visitors will be furnished to 

the institutions two months before the site review. 

3. The institutions will furnish, on request, information 
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required by the guidelines and by the site visitors. 

4. ihe Institution will provide site visitors with a 
structured presentation. The amount of time allotted for the 
presentat ion (s) will be determined by mutual agreement of the 
institution and the site visitors. 

5. Site visitors will have access to Institutional 
documents and Institutional personnel. 

6. Each site visitor will make i ; iependent judgments 
and independent reports on the visit. 

A feedback sess ion wi 1 1 occur at the end of each 
site review for the visitors and institutional personnel. Each 
visitor will make a tentative report and institutional personnel 
may question and correct the factual accuracy of the reports. 

8. After the site visit, each visitor will submit a 
draft report to the institution. Institutional personnel, then, 
will be able to respond to the site visitor regarding accuracy and 
Interpretation of the report*s content. 

9. After considering the institutional response, each 
site visitor will prepare a final report for government officials 
with an exact copy going to the institution. 

10. The institution may write a response to the final 
reports and send it to the authors and government officials. 

Evaluation Auditor 

The Advocate Team recommends t hat DRDR establish the evaluation auditing 
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process described below for the purpose of setting minimal levels of 
credibility for Its i ns t I tut i cna 1 reviews . 

The modes of evaluation discussed so far in this report suffer from a 
significant deficiency: they are carried out by OE personnel and/or 

their consultants* Consequently, critical reference groups such as the 
agencies being evaluated, the Department of HEW, and the Congress may 
doubt the evaluations credibility. 

The purpose of the evaluation auditing recommended here is 
to offset this possible credibility problem by providing for outside 
verification of the authentic! ty of the evaluation, findings and conclusions. 
Simply defined, evaluation auditing is superevaluation; it examines the 
processes and conclusions of lower-level evaluations for validity, 
reliability, and objectivity. 

The evaluation auditor in this process is analogous to the 
auditor in the business world. Generally, the latter fulfills two 
major functions: he verifies the authenticity of the accounting 

procedures of the firm being audited and the correctness of the 
accounting figures themselves. Potential investors then know the true 
fiscal position of the company and can be assured that its position is 
not being misrepresented. Similarly, investors, company managers, 
clients, and other publics can be reassured. 

The evaluation auditor provides similar assurance. He will 
certify after proper examination that DRDR and its contractors have 
considered a variety of value perspectives, have followed proper 
evaluation procedures and have ascertained findings and conclusions 
that are warranted by the collected data. Consumers of the contractor^ 
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r & d products; investors in those contracts (the Congress); DRDR, 
other branches of 0E, and HEW, who carry the legal responsibility for 
contract manageirien t ; the contractors themselves; and other publics can be 
assured that the evaluation reports essentially represent the true 
state of af fa ; rs . 

The evaluation auditor, of course, must be a person of 
unimpeachable character. He must possess technical expertise in 
evaluation plus substantive expertise in the area evaluated. For 
increased credibility, auditors will be recruited from outside of the 
education establishment. This will avoid the possibility of professional 
cooptation or the fear of i t on the part of noneducationist audiences. 
(Examples of such credible auditors include Michael Scriven, Phillip 
Clark, Paul Lazarsfeld, Lee Cronbach, and Stephen Bailey.) 

A group of potential auditors will be identified through 
nomination. The persons initially identified will be contacted for 
additional recommendations. The process will be repeated until a 
pool of potential auditors is formed. Each then will be approached 
for willingness to serve and as institutional proposals come up for 
review, DRDR will select from the panel. 

The processes and findings that the auditor may wish to 
examine in such a DRDR- i ns t i tu t ion evaluation interaction are described 
below. 

Identification of expert judges (field readers and site 
v j s i tors) . The auditor must assure himself that panels of expert 
judges (field readers and site visitors) were selected on the basis of 
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their qualifications and were assigned to particular institutions in 
non-biased (systematic) ways. Every person qualified to serve as a 
field reader and/or site visitor will have an equal chance of being 
i mpane lied. 

QE check review . The auditor must assure himself that the 
initial screening of institutional proposals by OE staff members was 
systematic and scrupulously conducted in accordance with published 
guidelines. He must examine the resulting analysis (which either 
recycles the proposal for rewriting or accepts the proposal for 
field-reader review) to certify the validity of the conclusion. 

OE veri f i cat ion . The auditor must assure himself that the 
verification of documents, facts, letters of support, etc., is presented 

f 

in the proposal objectively and systematically and that the degree of 

verification made is in fact proper. 

Field-reader r ev' _r ' e auditor must assure himself 

that the review of the r was systematically conducted 

and that his conclusions ere valid, replicable, and objective. 

Field-reader analysis ranking . The auditor must assure himself 
that the ranking made of the proposal in comparison with other proposals 
being simultaneously evaluated (and being audited by other evaluation 
auditors) was fairly and systematically accomplished. 

Site reviews . The auditor must assure himself that the site 
reviews were conducted systematically and that the site visitors 
conclusions ar« valid and objective. 

Site review analysis ranking . The auditor -must satisfy himself 
that the ranking made of the proposal in comparison with other proposals 
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being simultaneously evaluated (rind being audited by other 
evaluation auditors) was fairly and systematically accomplished. 

Certain expectations wi 1 1 be held for the auditor's 
performance. First, he will be engaged prior to the initiation of 
the rev i ew p roce ss and \ 1 1 be involved with it throughout. (He 
will establish feedback channels so that continuous refinements in 
the evaluation occur if unforeseen constraints, limitations, or 
problems are discovered.) Second, the evaluation auditor will not 
take responsibility for the evaluation process itself; rather, he 
will attempt to determine reasons for its success or failure at every 
step. His experience, consequently, may be gene ra 1 i zab 1 e to other 
evaluations to enable the whole process to be refined. Third, as 
indicated earlier, the evaluation auditor will perform roughly the 
same functions as his counterpart in the business world: certify 

that the methods of evaluation used fall within the limits of 
accepted evaluation practice and underwrite the conclusions and 
interpretations ar reasonable in terms of normal standards. He will 
examine all conclusions and interpretations to insure that they 
follow from the available data and are consistent with ordinary 
principles of logic and prudence. Fourth, the evaluation auditor 
will work on behalf of both DRDR and the proposing institution, 
pointing out difficulties each created for the other. Fifth, the 
evaluation auditor will make every effort to determine what decisions 
his reports will influence and to shape them so that they best service 
those decisions. Finally, the evaluation auditor will serve as an 
appeal agent in cases where the institution or DRDR believes that 
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the evaluation process was handled unfairly or mismanaged by one or 
more of the evaluation agents (DRDR evaluators, field readers, site 
visitors, ranking analysts, etc.). 

The auditor will not be responsible for contract or proposal 
approvals or for actions taken either by DRDR or by contracting 
i n s t i t u t i ons . 

To accompl i sh his functions he must have the right of 
complete and continuous access to all proposal data, reports, 
memoranda, and other records. He must be able to interview anyone 
associated with the proposal, whenever he considers it necessary; 
it must be his option to verify and validate any data or findings 
with their sources. He must receive copies of all correspondence, 
reports, and other key documents that relate to his responsibility. 
Finally, ne must be given a sufficient budget (to use at his sole 
discretion) to perform his duties. 

In his evaluation interactions, the auditor must be 
cautious to avoid cooptation by either party. The auditor must be 
close enough to the proposal review process to judge soundly its 
authenticity; at the same time, he must be distant enough to retiuer 
those judgments objectively. In particular, the auditor will have 
to avoid being used as a consultant by either or both parties. His 
constant interaction with them will make him highly susceptible to 
such use and his charge to render continuous feedback for refining 
and redirecting the process also will make him vulnerable. To 
guard against this, the auditor must document his continuing 
objectivity. If he is suspected of the slightest twinge of cooptation, 
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his credibility will vanish and the entire auditing process will become 
use 1 ess. 

DRDR wi 1 1 be involved in the evaluation of many proposals, 
particularly during the first few years of the new "mature i ns t i tu t i on 1 ' 
po 1 icy; howe ve r , the emp 1 oymen t of a full-scale audit f o r each 
evaluation is not feasible. Eco nom i c cons i derations militate against 
it, as does the scarcity of persons who might be used in evaluation- 
auditor roles. Moreover, a ful 1-scale audit is unnecessary, 
particularly if the primary intent is to maintain overall credibility 
rather than to insure absolute justice in all cases. Thus the use of 
auditing procedures on a p rob abalistic (s amp 1 i ng ) basis is justified. 
Taxpayers are kept honest by the possibility 1 1 *‘t their income tax 
returns may be audited at any time; similarly, valuations can be 
kept authent i c in the face of the possibility of an evaluation audit. 
Hence, while technically auditors are assigned to all evaluations, in 
fact they will function in only about 20 percent of all cases unless 
asked to intervene as appeal agents. Moreover, the 20 percent sample 
does not imply that 20 percent of all evaluations are sampled, but 
different segments of different evaluations (a verfication here, a 
site review there) so that the auditor's total work-load amounts to 
about 20 percent of all activities assigned to his jurisdiction. At 
the same time, the total range of evaluations for which some segment is 
sampled mav approach 100 percent of all evaluations. 
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VI 

Description of the System 

To see how the proposed model for evaluation and decision making 
will -appear under conditions of full implementation requires that 
a complex set of factors be considered from several perspectives. 
Consequently, this chapter describes the different roles that must 
be implemented by the involved institutions; an organizational 
structure for evaluation in DRDR; and a projection of cost factors 
necessary to calculate evaluation budgets for given circumstances 
and work loads. The material in this section is not conclusive: 
specific evaluation and decision-making work loads vary from year 
to year; consequently, they cannot be projected with any degree of 
confidence. For this reason, chapter six attempts to convey what 
would apply given specified assumptions about work loads in a given 
yea r . 
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I nst i tut i ona 1 Rol es 




Seven groups share the respons i b i 1 i t i es associated with implementing 
the tasks renui red by the proposed model: the DRDR decision-makers; 

the DRDR evaluators; the laboratory and center decision-makers; the 
laboratory and center evaluators; the external -expert judges; the 
external, techn i ca 1- se rv i ce agencies; and the external evaluation- 
auditors, This section defines roles for each group according to 
the requirements of the model presented in chapter two. The breakout 
of these roles serves as an organizing device. Where practicable, 
the tasks specified within roles are cross referenced to show how 
they might apply differentially to the t r 'n decisions and to suggestions 
that appear elsewhere in the report as to how the tasks might be 
performed. Also included are comments regarding the working level 
at which the tasks might be performed. 

DRDR dec i s ion-makers . DRDR decision-makers must assume 
the role of eva 1 ua t i on-o r i en ted leaders if the proposed system is to 
function effectively. They must orient their evaluators lo 
decisions to be served by evaluation and to the associated information 
requirements. They must seek to use evaluative data in arriving at 
decisions. Finally, they must seek and value the assessments of 
outside auditors concerning the extent to which decisions are based 
on sound information. Evaluation must be done proactively and 
systematically if decisions are to be efficiently served by a sound 
base of timely evaluation data; therefore, DRDR decision-makers must 
specify their information requirements in advance and insist that the 
DRDR evaluation system respond to these requests. 
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The eight tasks to be performed by DRDR decision-makers in 
implementing proposed evaluation and decision-making system are 
discussed be 1 ow . 

__ specification of decisions . Decisions are made at three 
basic levels within DRDR: division (or above), branch chiefs, 

and operational. The proposed evaluation system will provide 
information to assist in decision making at each of these 
levels. Thus DRDR persons must project the decisions for 
which they will need information in precise terms and in 
advance of when the information will be needed. Specifications 
of such decisions incHude what questions will have to be 
answered, by what agents, using what information, and at 
what future time. Such project will be made annually with 
respect to each of the ten deci s ion- types and to the breakout 
of decision-making responsibilities across the main levels of 

DRDR . 

The Offi ce of t h *’■ r) 1 ’ ' 1 r 1 on D i , LO r * f ^ «' '» / ■ - * * / 

prepare and distribute throughout the division on a document 
entitled "DRDR Annual Projection of Decisions and Associated 
Information Requirements." This document will be broken down 
accorcing to the ten dec i s i on- types -- each narrowed further 
regarding decision-making responsibilities at the main levels. 
Such document will be the main reference point for the design 
and i ro ' ementa t i on of D-1DR evaluation activities. 

-- Prov s i on to notify institutions of guidelines for the submit: 
of proposals and reports . In accordance with DRDR's decision- 
calenda-, r & d institutions will be notified two months before 
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future funding opportunities are announced and required 
evaluation reports are due. This information will be mailed 
to the institutions having financial support from DRDR. If 
possible, such information will take the form of specific 
guidelines such as those included in the appendix A. When 
nevj funding opportunities arise, agencies known to respond 
to requests for proposals will be notified by mail. In 
addition, widespread notice of the availability of guidelines 
for the preparation of proposals will be provided for other 
agencies. All agencies will be given two months to respond. 

-- Spec i f i ca t i on of dates for submission of proposals . 

In accordance with the cverall dec i 5 i on- ca 1 enda r and in relation 
to mailing out RFPs, d i v i s i on- 1 eve 1 decision-makers will 
specify deadlines for submitting funding proposals. Operational- 
level decision-makers, under the supervision of branch chiefs, 
must then make sure that all concerned parties are aware of 
th^se dates. 

-- Determination of the tentative acceptability of new 
proposals and reports . 0 pe ra t i ona 1 - 1 eve 1 dec i s i on-make rs will 
determine whether new proposals or reports contain sufficiently 
complete and accurate information for further consideration in 
the approval process. If the report fails this check, it will 
be returned to the institution for revision. The institution 
thus notified, may resubmit its report for further consideration 
if the noted deficiencies are rectified within one week. 

- - Decisions concerning the acceptance of proposals . 

Selection of proposals presumably will be made, at the level of 
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division director or above. Assistance from branch- arvd 
operational — level per sonne 1 generally will be required to 
develop tentative rank-orderings of the proposals or reports 
based on the data provided from the evaluation system. All 
decision-makers need to understand thoroughly the evaluation 
data; also, d i v i s i on- 1 eve 1 decision-makers at this point 
must consider that the external-evaluation auditors will 
aid the decision process by commenting as early as possible 
on the validity, reliability, and objectivity of the data. 
Decision makers at this step must seek to apply the criteria 
specified in the proposal and report guidel ines. At this 
point, proposals and reports will be ran ked in terms of their 
merits, without considering the availability of funds. 

-- Fu nding decisions . D i v i s i on- 1 eve 1 decision-makers 
will de ve lop explicit recommend a t ions con cerning the funding 
of proposals or the termination of funding of existing 
institutions or programs. Actual funding decisions, however, 
presumably will occur at levels higher than the division 
director. Funding recommendations must be justified in 
accordance with assumptions about budgetary constraints and 
based on previous rank- orde r i ng of the proposals and/or 
reports . 

-- Notification of applications regarding approval or 
disapproval o f proposals . Actual notification tasks probably 
will be performed by ope ra t iona 1 - 1 eve 1 personnel under the 
supervision of branch officers. However, personnel at the 
level of the director of the division probably will officially 
approve and sign the notifications. Institutions will be told 
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the ranking of their proposals and reports and the associated 
funding decisions. The basis on wh i ch the decisions were 
reached also will be described. 

-- Selection of expert judges . The pool of judges 

to read and rank proposals and participate in site visits must 

be selected at the level of the division director or above. 

The division director, with assistance from his branch chiefs 
and operational staff, will determine qualifications for 
membership in the pool and will select individuals in 
accordance with explicit procedures. Also, an outside agency 
(e.g.. Educational Testing Service) might continually work 
on the refinement of selection procedures as well as the 
implementation of the associated data gathering. 



DR DR eva 1 ua to rs . A strong office of evaluation must 
be established within DRDR to serve six functions. The first is to 
work con t i nuously w i th DRDR to delineate the information requirements 
associated with its annual decision-calendar. The second is to plan 
and coordinate the implementation of the basic da ta- ga t he r i ng system that 
services DRDR decision-makers. The third is to interpret and supply 
data to DRDR decision-makers regarding decisions. The fourth is to 
verify the accuracy of evaluative data submitted by institutional 
personnel to DRDR. The fifth is to provide liaison and coordination 
between DRDR and the external evaluation agents (i.e., the external 
expert judges, the external techn ical- service agencies, and the evaluation 
auditors). The sixth is to inform the external auditors concerning 
decisions reached in DRDR and the. eva 1 ua t i ve bases for those decisions. 
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The role of the DRDR. evaluators also encompasses 
specific tasks they are to perform within the framework of 
mode 1 . 



the th i r teen 
the proposed 




-- Development of guidelines for proposals and evaluation 
reports . Institutions must sys tema t : ca 1 1 y provide basic 
data to assist DRDR decision-makers in making all decisions 
wi th the exceptions of 7, 8, and 10. In the case of 7, 
institutional personnel operating existing programs wi 1 1 
design and conduct their own evaluations with only general 
guidance from DRDR. Evaluations to support decision 8 
are ad hoc and emerge only in response to special circumstances, 
i.e., concern tfiat an existing program within a mature institution 
is jeopardized because several of its key program staff are 
leaving. Evaluations to support decision 10 will be based on 
basic data from the field, usually derived from existing 
evaluation reports presented to DRDR in reference to other 
decision types . 

DRDR evaluators -- probably intermediate master's- 
level personnel in the evaluation office -- will prepare guidelines 
based on information in the annual DRDR projection of decisions 
and associated information requirements similar to the guidelines 
presented in the appendix A. It is suggested that DRDR begin 
by using these suggested guidelines and that thereafter the 
guidelines be reviewed and updated annually. Also, DRDR 
evaluators must work continually with outside technical 
agencies that have contracted to do data analysis to improve 
the da ta- ga the r i ng guidelines. 
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- - Development of guidelines and instruments for the reading 
of proposals and for site reviews . The same conditions as 
explained for proposals and evaluation reports apply here. 

- - Check for completeness of reports and proposals . 

Pa raprof ess i ona 1 personnel »n the evaluation office will 
systematically check all reports and proposals received to 
insure that they provide necessary information. 

This task generally will be heavy and contirxjous; 
however, peak loads can be projected by reference to the DRDR 
annual projection of decisions and associated information- 
requirement documents. The main concern will be for decisions 1 - 6, 
since these all have funding implications and specific time 
constraints. Under a less rigorous schedule the paraprof ess i ona 1 s 
will also check data provided by the institutions in reference 
to decisions 7 and 9- 

Training sessions for the pa rap rof ess i ona 1 s probably 
will be conducted by intermediate (master^ level) personnel 
in the evaluation office. The guidelines in appendix A will 
help design such training activities. 

-- Development of reports concerning the validity of proposals 
and evaluation reports . Intermediate (master's level) 
personnel will be involved in determining the truth value 
of data provided by the institutions. Such data generally 
lack objectivity and cannot be assumed to be credible to 
the DRDR decision-makers; consequently, validity checks are 
crucial, especially regarding decisions 2, 4, 6, and 9. For 
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these decisions the i n te rmed i a te- 1 eve 1 DRDR evaluators will 
make site visits or employ special consultants to conduct such 
visits. Procedures for these site visits must adhere to those 
specified in chapter five. 

Basically, the validity checks will begin by 
identifying the critical claims made in reports and proposals; 
then, the data provided will be checked to determine whether 
they support the claims. Next, the proposals and reports will 
be checked for accuracy of the data. Finally, if there is 
any question and if the decision is considered to be a 
significant one, (especially including 2, 4, and 6), site 
visits must be carried out and final reports written. The 
basic step within this role is determining the significant 
claims in the proposals and reports. This will be done by 
referring to the guidelines that were the basis for preparation 
of the proposals and reports and to the rank order of criteria 
that formed the basis for the guidelines. 

The i ntermed i ate- 1 evel personnel in DRDR will be 
trained specifically to perform this role. Such training 
will include simulated validation checks of proposals and 
reports for decisions 2, 4, 6, and 9 and participation 
as an observer in site visits. Such training might be 
subcontrac ted to an agency that trains evaluators (e.g.. The 
Ohio State University, University of Colorado, or University 
of Illinois). An> such contract should call for the 
preparation of appropriate s imu 1 at ion- t ra i n i ng materials 
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that coi’ld be used within DRDR on a continual basis following 
the Initial project. 

-- Coordination and monitoring of the work of expert judges . 

I n te rmed i a te- 1 eve 1 personnel in DRDR will implement the 
specified procedure (see chapter five) for obtaining data 
from expert judges concerning evaluation reports and proposals 
and site visits. Accordingly, the DRDR evaluators must assign 
judges to specific reading and/or site visit tasks; orient 
the persons to their tasks; schedule and arrange the conditions 
required for the judges to implement their roles; coordinate 
the work of the judges; secure their reports; organize and 
store the data provided by the judges; and ma^e the basic 
data available to the external agency that will perform 
the data analysis. The activities relating to reading of 
proposals and reports apply especially to decisions 1, 3, and 5; 
while the role regarding site visits pertains more to decisions 
2, 4, 6, and 8. It has been suggested elsewhere in this 
report that site visitors must be the same persons who read 
previous proposals and reports associated with the same 
programs or institutions. Normally, then, the sequence of 
the subtasks will call for reading and then site visiting. 

Therefore, scheduling of the reading and the site 
visiting must be done as far in advance as possible and within 
the overall dec i s i on- ca 1 enda r contained in the "DRDR Annual 
Projection of Decisions and Associated Information Requirements. 11 
The i n te rmed i a t e- 1 eve 1 evaluators responsible for the coordination 
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of reading and site-visit tasks must adhere as closely as 
possible to the procedures specified in chapter five pertaining 
to assignment of judges to tasks and the conduct of site visits. 
The evaluators also must become thorough 1 y f am i 1 i a r with the 
contents of guidelines that appear in appendix A. 

Coordination and monitoring of the work of the outside 
analysis system . Data derived from expert judges regarding 
both the reading of reports and proposals and site visits 
will be submitted to a thorough statistical analysis to 
arrive at a rank ordering of the proposals and reports. 
(Procedures for such analysis, provided in chapter five, are 
to be administered by an external technical services agency 
_/e.g., Educational Testing Service/.) 

For such an agency to perform its role satisfactorily, 
it must be monitored and serviced directly by DRDR evaluators. 

An i n termed i ate- 1 eve 1 evaluator from DRDR can perform the 
1 i nkage function. Th i s eva luator will delineate an analysis 
schedule in co 1 1 abora t i on with personnel from the technical- 
services institution and will provide data to the agency and 
retries its report in accordance with the specified schedule. 
Such scheduling should be done annually based on the "DRDR 
Annual Projection of Decisions and Associated Information 
Requi rements" document. 

Also, the liaison evaluator must seek to coordinate 
efforts to improve both the da ta- ga the r i ng instruments and 
the analysis procedures. Activities within this area will be 
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most concerned with decisions 2, k, and 6 because they require 
the most thorough analysis and rank ordering. The analysis 
agency also will service decisions 1, 3, and 5, hut not with 
the same urgency and criticality as the three previously 
mentioned decisions. 

-- Interpretation and reporting of rank-order data prov ided 
by the external analysis system. The i n t e rmed i a te- 1 eve 1 DRDR 
evaluation, serving a liaison function between the DRDR 
eva 1 ua t i on- of f i ce and the external techn i ca 1 - se rv i ces agency 
responsible for analyzing data to rank order proposals and 
reports, will secure reports from the agency and distribute 
them to other i n te rmed i a te- 1 eve 1 evaluation personnel 
responsible for servicing the evaluation requirement 
associated with particular decisions. These latter intermediate- 
level evaluators then will prepare a staff memo summarizing 
and evaluating the accuracy of the rankings that have been 
derived by the analysis agency. An evaluator's report also 
will indicate what decisions tend to be indicated, given the 
Specific conditions of the decision situation and the data- 
analysis results. The evaluator then will convey his written 
report to the top-level evaluator in his office and will 
make modifications that both parties agree to. Any 
irresolvable conflicts regarding the contents of the report 
will be noted in the report, rather than compromised. Next, 
the i n termed i a te- 1 eve 1 evaluator in charge of servicing the 
given decision will submit his report both orally and in 
written form to the designated dec i s i on- aud i ences . 
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The division director and decision makers at higher 
levels will comprise the primary audience for reports that 
serve decision types 2, k, 6, and 9. The reports that service 
decisions 1, 3, 5, and 10 should be aimed most directly 3t 

branch-level decision-makers. The timing for the preparation 
and presentation of these reports must be clearly projected 
on an annual basis in accordance with the "DRDR Annual Projection 
of Decisions and Associated Information Requirements. 11 
Moreover, the director of the evaluation office must consider 
it one of his prime responsibilities to delineate the differentia] 
responsibilities of his i ntermed i a te- 1 eve 1 evaluators with 
respect to the decision situations that they are to serve. 

The i ntermed iate- 1 evel evaluators, who interpret and 
make recommendations based on reports received from the external 
analysis agency, must have a working knowledge of advanced 
statistics (e.g., statistical concepts such as those covered 
in B. J . Winer's Statistical Principles of Experimental 
Desig n and j. P. Guilford's Psychometric Metho ds ) . In 
making their reports these evaluators must be able to critique 
the rank ordering of proposals or reports provided by the 
outside firm in terms of reliability and validity. They 
also must make recommendations regarding the approval of 
reports and the acceptance of proposals in terms of the 
original guidelines, specified funding constraints, and data 
provided by the outside analysis firm. Personnel to serve in 
these roles might well be recruited from among mas te r ' s- deg ree 
graduates in mathematical statistics, economics, or industrial 
psycho 1 ogy . 
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L'aison with the external evaluation auditors. One or 
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more i n te rmed i ate- 1 eve ] evaluators will be assigned as full 
time liaison with the external evaluation auditors. This 
evaluation specialist w i 1 1 be distinct from the ones who 
provide liaison with the outside analysis agencv and who 
service particular decisions. The exte rna 1 - au d i t o r liaison 
evaluation-specialist will assign evaluation auditors to 
various decisions to be serviced in accordance with specified 
procedures. These auditors will come from a pool of personnel 
selected by the division director. The audit 1 i a i son- 
specialist will orient ihe auditor to his responsibilities; 
supply him with relevant guidelines, evaluation reports, 
and accounts of decisions reached in DRDR ; and assist him to 
communicate his audit reports to relevant audiences. 

As noted in chapter five, the DRDR eva 1 ua t i on- of f i ce 
will implement an external audit. Procedures must be specific 
for selecting the pool, assigning the auditors to specific 
decisions to be serviced, securing specified reports from 
them, providing them with access to key decision-makers whenever 
they believe an intervention is warranted, and insuring that 
audit reports will reach appropriate audiences. As the DRDR 
office of evaluation confronts this task, it needs to draw 
heavily on the experience developed in the Title VII and 
Title VIII programs of the Elementary and Secondary Education 
Act of 1965 in the Bureau of Elementary and Secondary Education. 
Consultants might expedite DRDR's development of a system to 
provide adequate external audits for its evaluation and decision- 
making systems. 
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~~ Assignment of judges to reading and s i te-v i s i t ^tasks L . 

Intermed ate-level evaluators responsible for providing 
recommendations regarding specific decisions will a . s*. be 
responsible for assigning judges to read ing and site- Visit 
tasks when necessary. And these I n te rmed i ate- 1 evel 
evaluators wi 1 1 arrange appropriate site visits. 

The assignment of site visitors will be in 
accordance with procedures specified in chapter five. This 
selection wi 1 1 occur annual ly as soon as the n DRDR Annual 
Projection of Decisions and Associated Information Requirements" 
has been completed. 

— Orientation and instr ucti on of the expe r t_j udges . The 
i n termed i ate- 1 evel evaluator who assigned the expert judges to 
their tasks must also insure that they meet their responsibilities. 
This will include orientation to specific reading and site-visit 
procedures (see chapter five), explication of the decision 
situation to be serviced, assignment of documents to be read 
and sites to be visited, and provision of guidelines and instruments 
by which the judges will perform their rating function. 

Reading assignments will be made especially in 
reference to decisions 1, 3, and 5, while site-visit assignments 
will be made especially in relation to decisions 2, 4 , and 6 . 

Either or both the reading and site-visit procedures may be 
applied, based upon DRDR decision makers 1 request, to decisions 
7, 8, and 9. The timing of the reading and site-visit 
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assignments will be projected in accordance with the ,! DRDR 
Annual Projection of Decisions and Associated Information 
Requ i rement s . 1 1 

-- Preparation of institutions for site reviews . The same 
i ntermed i ate- 1 eve 1 DRDR evaluators w ho made site-review 
assignments will assist the institutions to be visited. These 
institutions will be presented with the basic criteria to be 
applied by DRDR decision-makers in reaching a decision 
pe.talning to the institutions, names of site reviewers, 
guidelines to use in providing data to the site reviewers, 
the instruments the site reviewers w i 1 1 be using in applying 
the specified criteria, a suggested set of dates for the site 
reviews, as well as a time when the decisions that are the 
subject of the site reviews must be made. Guidelines for 
site visits will be provided to the institution at least 
six months prior to the projected decision-date* 

-- Coordination of site reviews . The intermediate evaluators 
who arrange site reviews will observe them and assist in their 
conduct according to the specified procedures (see chapter five)* 
And especially, the i ntermed i ate- 1 eve 1 evaluator will monitor 
the site-visit proceedings to insure that they meet their 
ob j ect i ves . 

The above discussion of tasks indicates that DRDR must have 
three levels of evaluation staff: administrative, intermediate, and 

pa rap rof es s I onal * Further, the i nte rmed i ate- 1 eve 1 staff must serve four 
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distinct areas of responsibility: maintaining liaison with the 

external t schn i ca 1 - se rv i ce agencies; obtaining basic data: analyzing 
reports from the techn i cal - servi ces agencies; and maintaining liaison 
with the external auditor. (Later in this chapter an organizational 
breakdown will be provided for the staffing of the DRDR evaluation 
of f i ce. ) 



Expe r t judges . As discussed in chapter five, the 
proposed evaluation and decision-making system will depend heavily for 
evaluative data on a large, well-defined and specially qualified group 
of expert judges. Such judges will be selected for their qualifications 
that relate to given strata associated with criteria to be applied in the 
ten dec i s i on- s i tuat i ons . 

Two basic strata are to be considered here. The first pertains 
to institutional criteria; the second applies to the substance of the 
program. The breakdown of criteria elsewhere in this report for programs 
versus institutions helps classify the qual i f i cat i ons of the expert 
judges. After judges have been selected and sys temat i ca 1 1 y assigned within 
strata, they will be assigned to specificed review-tasks according to the 
systematic procedure specified in chapter five. After assignment, the 
judges will adhere to the specified procedures for reviewing proposals and 
reports and for pa rt i c i pat i n g in site reviews. Essentially, they will 
produce rankings of institutions 1 programs and/or proposals in terms of 
specified criteria, given guidelines, and instruments. (Suggested 
guidelines and instruments appear In appendix A.) The output of the 
expert judges, therefore, is not to be taken as the complete judgmental 
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job, but rather as the provision of judgmental data that wi 1 1 be further 
analyzed (by a specified outside techn i ca 1 - serv i ces agency) before 
rank-order decisions can be made by DRDR officials. 

Four specific tasks of the expert judges can be derived 
from the proposed model for evaluation and decision making. 

-- Judgments of proposals and reports . As specified in chapter 
five, judges - after assignment to reading tasks - will 
receive and read proposals and reports and then rate them 
according to the guidelines that appear in appendix A. This 
rating responsibility pertains especially to decisions 1 - 6 
and 8. Scheduling of the distribution of reports and obtaining 
of rankings from the judges will be projected annually in 
accordance with the n DRDR Annual Projection of Decisions and 
Associated Information Requirements. 11 

-- Eva 1 uat ion reports . In accordance with specified guidelines, 
expert judges will prepare and submit reports to DRDR that 
contain the judges 1 findings concerning the relative strengths 
and weaknesses of the reports and proposals. Such reporting will 
be done in accordance with the requirements of the structured 
guidelines presented in appendix A. 

-- S i te rev i ews . The expert judges will participate in site 
reviews according to the procedures specified in chapter five. 
Also, it is recommended that an outside service agency (e. g. , 
Educational Testing Service) continually work on the refinement 
of selection procedures as well as the implementation of the 
associated basic data- gat her i ng. Site reviewers for given 
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agencies will be the judges that have read the proposals 
or reports that preceded the site visit. 

“ Site-review reports . Judges wall prepare and submit 
reports based on their site reviews and previous document- 
reading in accordance with the chapter five procedures 
and the appropriate guidelines for both the reading and site 
visits conducted. These reports, usually pertaining to 
decisions 2, 4, and 6, will provide explicit ranks for the 
ma i n criteria noted in the report guidelines. When possible, 
the site visitors will provide comments that support the 
given rank. The visitors also must realize that their ranks 
will be tested with the ranks of other independent judges^ 
for reliability and validity and that such information will 
be taken into account, along with the judges' ranks, in 
the making of decisions. (Judges, throughout their site 
review and reading roles, must act independently rather than 
reaching a consensus with other judges.) 

External techn i ca 1 - se rv i ces agency . DRDR does not have the 
analysis capability and technical expertise required for instrument 
development, the development of complex analytic procedures required 
for rank ordering of proposals and reports, and the implementation 
of complex analysis procedures. Therefore, it is suggested that 
outside agencies (e.g., Educational Testing Service, Resource 
Management Corporation, and the Iowa Educational Information Center) 
be given both continuing and short-term contracts for the development/ 
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refinement, and implementation of data gathering and analytic 
procedures that can result in rel i ab 1 e ranking of reports. Three 
specific tasks are apparent in the proposed model for evaluation and 
decision making that must be performed by an outside agency. 

— Development of analytic procedures . Chapter five suggests 
analytic procedures to be followed in ran k- o r de r i ng proposals 
and reports. However, development and refinement of such 
procedures must be a continuing process, especially with 
re r pect to continually seeking to provide more reliable and 
valid instruments. An institution might well provide systematic 
service to DRDR for the development of sound da t a- ga t he r i n g 
instruments and procedures for i mp 1 emen t a t i on of the proposed 
evaluation and decision-making system. Also, as experience 
is gained with the instruments and procedures suggested in this 
report, the analysis of data can be used not only to help 
DRDR but also to assist in the refinement of the Instruments 
used to gather the original data. Thus refinement of 
procedures could be a routine side-benefit. 

Development of a pool of expert judges . Institutions with 
techn i ca 1 - se rv i ce capabilities will be engaged to identify 
and propose the pool of expert judges. Further, such 
institutions could develop a procedure for sy s temat i ca 1 1 y 
defining the panel o r expert judges in such a way that 
criteria gleaned from guidelines for ranking proposals and 
reports and judging institutions and programs based on site 
visits could be employed to select judges from appropriate 
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strata within the overall pool of judges. This is a vital 
role to be performed; consequently, it is urged that DRDR 
let an outside contract for performance of these tasks. 

-- Conduct of special analyses for external auditors . Since 
the proposed evaluation and decision-making system requires 
the involvement of external auditors, it is anticipated that 
auditors at times will question the accuracy or completeness 
of analyses performed by the external techn i ca 1 - se rv i ce s 
agencies in respect to the ranking of reports and proposals. 
The auditor may call for a special kind of analysis or even a 
reanalysis report. Both the original analyses and the 
reanalysis r epo r t are mos t 1 i ke ly to occur with respect 
to decision types 2, 4, and 6, 

Evaluation aud i tors . The role of the evaluation auditor 
is defined in chapter five. Auditing assignments are made by the 
DRDR decision-makers; consequently, the auditors must have continuing 
access to DRDR data and personnel to meet their responsibilities. 
Generally the auditor should be encouraged to make audit exception 
reports at any time. And more than that, the DRDR evaluation specialist 
will structure the auditor 1 s role so that systematic reporting can be 
done after the submission of evaluation reports by the analysis agency 
and following decisions reached in DRDR. No specific task for the 
auditor appears within the proposed model for evaluation and decision 
making since the auditor is supposed to monitor the entire evaluation 
and decision-making process. The technical procedure for selecting and 
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assigning auditors to serve specified evaluation needs to be worked 
out. However, chapter five recommends a process that DRDR might follow 
in designing and implementing its audit system. 

Institutional decision-makers . Institutional decision 
makers obviously play a key role in the proposed evaluation and decision- 
making system. They prepare and submit the basic proposals and reports 
for most DRDR decisions; they maintain internal evaluation mechanisms 
that supply the data contained in the proposals and reports; and 
finally, they assist in the coordination of site reviews. Institutional 
decision makers assumedly include the directors of' the institutions 
as well as their designated representatives. 

-- Proposals and reports . Institutional decision-makers not 
presently administering DRDR funds may respond to special 
requests for proposals sent out by DRDR, as in the case of 
decisions 1 and 5. Institutional decision makers already 
possessing funds from DRDR must respond to its requests 
for reports concerning presently funded activities and may 
respond to requests for proposals for the allocation of further 
fund s. 

-- Resubmission of reports or proposals . When DRDR evaluators 
or decision makers find that the information provided in 
required proposals or reports to DRDR is inadequate, the 
institutional decision-makers wh o submitted the proposals or 
reports must rectify the deficiencies and resubmit them. This 
is mandatory in the case of required reports and voluntary in 
the case of proposals. As noted in chapter two, proposals 
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first judged incomplete or inadequate will not be considered 
for funding until information deficiencies in those documents 
are rectified by submitting institutions. The institutional 
decision maker's responsibilities in the area of reports and 
proposals occur mainly with respect to decisions 1 - 6, and 9. 

Coordination of the site review . Decisions 2, 4, and 6 require 
institutional decision makers to work with DRDR evaluators 
in planning and conducting site reviews. The reviews will 
be mainly under the direction of DRDR. Also, DRDR decision 
makers will organize and coordinate site reviews concerning 
decision 7. But in the latter situation, institutional 
decision makers will be in charge of organizing and conducting 
the site rev i ews . 

Institutional evaluators . Evaluation offices and personnel 
within agencies supported by DRDR, as well as those with potential DRDR 
support, play a fundamental role in the proposed evaluation and decision- 
making system for the division. These evaluation agencies will supply 
most of the basic data required for the system’s implementation. 

It is expected that DRDR- supported institutions will possess 
evaluation capabilities in four main areas. First, they will be able to 
present data that support the objectives of the institution. Second, they 
will be able to present data that justify the procedural plans adopted for 
the achievement of institutional objectives. Third, they will be able to 
provide data that reflect the extent to which planned procedures are 
being implemented. And finally, they will be able to present data on the 
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extent to which the institution’s objectives are being achieved. Moreover, 
within this basic framework of information capabilities the institutional 
evaluators will be able to respond to the specific data requirements 
associated with decisions 1 - 7 and 9. 

Four specific tasks are observed in the proposed model for 
evaluation and decision making to be served by the institutional 
eva 1 ua to r s . 

-- Provision of data needed for proposals and reports . As 
already noted, the institutional evaluators will provide 
the basic data required for proposals and reports that pertain 
to decisions 1 - 7 and 9. These evaluation data must be 
in accordance with the associated sets of guidelines that 
appea r In appendix A. 

-- Provision of any missing information . When reports or 
proposals are returned from DRDR, the institution's 
decision maker will rely heavily on his internal evaluation- 
personnel for the rectification of the information deficiencies 
that had been identified in the submitted proposal or report. 

-- Preparation of data for site reviews . Institutional evaluators 
will prepare data to answer the basic questions that appear in 
the guidelines (as found in appendix A) for site reviews. The 
decisions of most importance are 2, 4, and 6. 

„ provision of data at the site review . During the conduct of 
site reviews for decisions 2, 4, 6, or 7 (since the institution will 
be in charge of site reviews for decision 7), the internal evaluator 
will need to be available to interpret data provided, substantiate 
its accuracy, and supply any missing components. 
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Since the proposed system for evaluation and deci s l on-mak i ng 
within DRDR depends so heavily on the submission of basic data from 
Institutions, DRDR must support strongly the establishment of evaluation 
offices In the institutions it sponsors. 

DRDR Evaluation Office 

In the preceding section numerous tasks were identified as being the 
respons i b i 1 i ty of personnel in the DRDR evaluation office. This section 
groups the eva 1 u a t i on tasks into several jobs within an overall 
o r gan i zat i ona 1 framework. 

Figure 3 indicates that the evaluation office of DRDR will be 
a staff office to the director of DRDR, with the associate director for 
evaluation (who heads the DRDR evaluation unit) reporting directly to the 
division director. An adm i n i s t rat i ve assistant serves in a staff relationship 
to the associate director for evaluation, while three kinds of specialists 
report in a line relationship to the associate director. The latter are 
the audit liaison specialists, the techn i ca 1 - se rv i ces 1 i a i son spec i a 1 i s ts , 
and the institution liaison specialists. Also, pa rap rof ess i ona 1 evaluators are 
shown to report to the adm i n i s t ra t i ve assistant. The three kinds of evaluation 
specialists have been identified so as to group all responsibilities in 
relationship to the client served. 
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F i gu re 3 

Proposed 0 r gan i za t i ona 1 Chart 




Existing DRDR 
Structure 

institution liaison specialist . Institution liaison specialists 
will perform all tasks associated with the gathering of data from field DRDR- 
supported institutions. These specialists will distribute guidelines for 
proposals and reports and plan and coordinate their reading by expert judges. 
Further, the specialists will assess the validity of proposals and evaluation 
reports submitted by institutions. The will inform institutions of criteria, 
names of site reviewers, da ta~ ga t he r i n g guidelines the site reviewers will be 
using, dates for the site visits, and dates for decisions to be reached. 

And they will insure that site visits and reading procedures are implemented 
app rop r i a te 1 y . 

Personnel performing this role must have a working knowledge of 
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measurement as we I I as strong organizational and i n te rpe rsona I - re 1 a t i ons 
skills. Persons with backgrounds in educational measurement and sociology 
should be given preference in the staffing of these roles. 

Audi t i iaison special ist . The audit liaison specialists' job will 
encompass all activities associated with providing liaison between the DRDR 
evaluation-office and the external auditors. These specialists will assign 
auditors to audit responsibilities in accordance with the procedures 
specified in chapter five; they will provide orientation to the auditors 
and will supply them with a continuing flow of pertinent data. They also 
will convey the auditors' reports to the appropriate personnel and will 

assist in cases where the auditor has asked for new analyses or reanalyses 
of data . 



Personnel with a good working knowledge of evaluation methodology 
would be most desirable for performance in this liaison role. 

Tec hnical -services liaison specialist . The job of the technical- 
services liaison specialist will encompass all of the activities associated 
with securing specially contracted, outside technical services to support the 
data collection and analysis associated with decisions 2, U, and 6. Persons 
in this role will work on the continual refinement of basic da ta- ga the r i ng 
instruments and the associated analysis procedures. They will seek to effect 
contracts with appropriate technical service agencies (e.g.. Educational 
Testing Service) and wi I I work with them to implement the bas i c data-ana I ys i s 
required for rank ordering of proposals and reports. Further, the technical- 
services liaison specialist will analyze data reports from the technical- 
services agency in the context of funding constraints and will provide 
recommendations to the associate director for evaluation and DRDR decision- 
makers regarding the approval of proposals and reports. 
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Personnel to implement this role should have a strong background 
in mathematical statistics and economics. While they probably need 
training only at the mas te r 1 s- degree level if their spec i a 1 i za t i on has 
been mathematical statistics, these people must be able to systematically 
supervise businesslike rel a t i onsh i ps with outside funding agencies. 

Associate director for evaluation . The associate director 
for evaluation will report directly to the division director and will 
work closely with him and other DRDR dec i s i on-make r s to insure that the 
evaluation system remains strong in its support of decisions. The role 
is both adm i n i s t ra t i ve and interface. Administratively the associate 
director must insure that his staff functions efficiently and effectively 
with respect to all of their evaluation responsibilities. Staff 
development will be an important concern; especially, he must insure that 
the pa r ap r of es s i o na 1 s and the three kinds of evaluation special ists 
receive special instruction with respect to implementation of their 
roles. Also, adequate training must be provided for the expert judges 
if they are to implement the data- ga t he r i ng methodology, continually 
under refinement. 

Admi n i s t ra t i ve ass istant. An administrative assistant will 
report to the associate director for evaluation. This assistant will 
attend to adm i n i s t r a t i ve details associated with efficient operation 
of the evaluation office and will supervise the work of pa r ap rof es s i ona 1 s . 

The person in this role might have a bachelor's degree in 
business admi n i s t rat I on . He needs a general working knowledge of 
evaluation procedures, but more importantly he needs skills in the areas 
of management and business practice. 
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Pa raprof es s i onal s . Paraprofess ional s wi 1 1 be assigned to 
assist the evaluation specialists. Pa rap rof es s i ona 1 s 1 tasks include 
checking reports and proposals for completeness against guidel ines, 
processing incoming and outgoing documents, coding data, and performing 
inhouse analysis tasks. 

A high-school education should be sufficient for adequate 
performance of this role. Regardless, the office of evaluation will 
provide the necessary training for a person of average intelligence 
to implement the role of the pa rap rof ess i ona 1 . 

Funding of the Proposed Evaluation System 

Tables 3 and 4 will aid DRDR decision-makers in projecting costs 
associated with implementing the proposed evaluation system. 

Table 3 shows an approximate unit number of man-days 
for each decision-situation. The time is based on allowing one 
day to read and critique a report, and three days are allowed to 
conduct a site visit, two days for the visit and one day for 
compiling a report. Columns (1) and (2) estimate the number of 
DRDR evaluation and DRDR-pa rap rof ess i ona 1 man-days required per 
proposal or report. Columns (3) and (4) show the number of readers 
and reader man-days required per proposal or report for each 
decision; these columns are identical since the reading of each 
proposal will require one day per judge. Columns (S) and (6) 
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Table 3 



Man-Days of DRDR Evaluators and 
Expert Judges Needed for Ten Decision Situations 



0) 
DRDR 
Profes- 
sional 
Eva 1 1 r . 



( 2 ) 
DRDR 
Para- 
Prof. 
Eva 1 t r . 



(3) 



Reade rs 



c*o 



Reade r 



(5) 



( 6 ) 



V ? s i tor 



Visitors 



(7) 
Total 
Visitor 
pi us 
Reader 
days 



1 


-5 


.25 


3 


3 




3 


2 


n 

J 


-50 


3 


3 


3* 9** 


12 


3 


3 


.75 


3 


3 


3 9 


12 


4 


4 


.75 


5 


5 


5 15 


20 


5 


1 


.50 


3 


3 




3 


6 


4 


.25 


5 


5 


5 15 


20 


7 


1 












8 


4 


-25 


5 


5 


5 15 


20 


9 


1 


.50 


3 


3 




3 


10 


.5 












Three 


s i te 


visitors 


are used for 


Dec i s i on 


2 s i tuat ions . 


Each 


is all owed two 


days for 


the visit and 


one day 


to produce a 


report . 


Some proposals 


may be e 


liminated after judges 


read the proposals; 


hence , a 


s i te 


rev i ew wi 


11 be unnecessary. The 


number of tri 


ps can be 


based on 


the number in 


the M Number of 


V i s i tors 


n column to calculate 



travel costs for judges. 

-**v Required only when warranted by special circumstances. 
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present the number of visitors and visitor days, assuming three days 
per visitor, per proposal or report. Column (7) estimates the 
combined visitor and reader days required per report or proposal 
for each of the ten decisions. Columns (l) and (7) reveal that 
decisions 4, 6, and 8 require the most extensive evaluations per 
decision alternative under consideration, while decisions 2 and 3 
f o 1 1 ow closely. 

For the figures in Table 3 to have practical meaning it 
is necessary to know the number of decision alternatives to be 
assessed for each decision type. For budgetary planning such 
projections must be made at least on an annual basis. In the 
proposed evaluation system, projections of the number of each type 
of decision to be made during a given year takes operational form 
in the n DRDR Annual Projection of Decisions and Associated Information 
Requirements' 1 . Given the number of each type of decision to be 
made during a given year, the contents of Table 3 can be used to 
project the number of DRDR evaluator and expert judge man-days 
requ i red . 

Table 4 illustrates how man-days can be projected for 

one year. Based upon discussions with a DRDR representative, 

estimates were made of the number of each type of decision that 

1 

will be made in DRDR during the next year. The estimates appear 
in Column (0) . Columns (1) - (7) estimate the DRDR evaluator 

^The precision of these estimates is not crucial here because 
they are being used only for i 1 lustrative purposes; if the proposed 
system becomes operational, such projections will be derived from the 
“DRDR Annual Projection of Decisions and Associated Information 
Requ i rements". 
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Table 4 



Man-Days for an Assumed Typical Year 



Dec i - 
s i ons 


(0) 
Esti- 
mated 
No. of 
Dec i - 
s i ons 


(0 

DRDR 
Eva 1 u- 
ator 
days 


(2) 

DRDR 
Para- 
Prof . 
days 


(3) (M (5) (6) 

No . of Visi- 

No. of Reader V i s i - tor 
Readers days tors days 


(7) 

Total 
Vis? tor 
and 

Reader 

days 


1 


50 


25 


12 


150 


150 






150 


2 


9 


27 


5 


27 


27 


27 


81 


108 


3 


3 


9 


2 


9 


9 


9 


27 


36 


4 


3 


12 


3 


15 


15 


15 


45 


60 


5 


30 


30 


15 


90 


90 






90 


6 


9 


36 


2 


45 


45 


45 


135 


180 


7 


75 


75 














8 


3 


12 


1 


15 


15 


15 


45 


60 


9 


10 


10 


5 


30 


30 






30 


10 


25 


13 


















249 


45 


381 


381 


1 1 1 


333 


714 


Note : 


Decision 7 should occur throughout 


the year; dec 


i s i on 10 


now occurs during two months; an effort should be made 


to schedule 


types 


1 , 2 , 5 , and 6 


at least three months apart 


during 


the year 


to avoid peak 


1 oads 


on DRDR 


evaluation staff 
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and expert judge man-days required per decision type, given the 
projections in column (0) . Entries in columns (1) - (7) of Table 

4 were calculated by multiplying the corresponding entr'es from 
columns (1) - (7) of Table 3 times the number of decisions entered 

in col umn (0) of Table 4. 

Table 4 assumed that fifty type 1-decisions will be made 
in a given year. These will require 25 man-days from DRDR 
professional evaluators, 12 man-days from the DRDR pa rap rof ess i ona 1 
evaluators, and 150 man-days from the expert judge readers. 
Accordingly, the remainder of the table can be read for each 
dec i s i on . 

In general, under the given assumptions, a full-time 
mas te r s- 1 eve 1 person can manage the i ns t i t u t i ona 1 - 1 i a i son role 
if the time is spread out linearly. If not, extra personnel will 
have to work in this category when peak loads exist. Since the 
assumptions dictate that the parap rofess iona 1 evaluator will work 
45 man-days, this role can be combined for the present with that 
of a full-time adm i n i s t rat i ve assistant. 

Additional evaluation staff will be required continually 
to develop and administer the new evaluation system and to assist 
DRDR to make use of the system's evaluative outputs. Also, the 
evaluation staff surely will be called upon to prepare summaries 
of data and papers in response to DRDR planning needs, to conduct 
special studies, and to answer requests from government offices 
and Cong ress . 
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Given the analysis of Table 4 and the additional core 
staff needs identified above, funds should be available for the 
f o 1 lowing: 



Staff 

1 d i rector 

1 administrative as s i s t ant/pa rap rofess Iona 1 evaluator 
1 institution liaison specialist 
1 technical services liaison specialist 

1 audit liaison specialist 

2 sec re t a r i es 

0 1 ne r 



714 days for judges ((<*> $150) 

30 days for clerical ((u> $30) 

Contract services for the development of instruments 
and analysis procedures 
Contract services for analysis (13 batches) 

Travel for judges (120 trips (a) $250) 

Other staff-support costs (e.g. travel) 

Contract services for developing the audit system 
Evaluation auditors (50 days annually (® $150) 

Training for 50 expert judges ($2,000) 

The analysis in Table 4 assumes a more efficient system 
than exists in a federal office. hactors such as slow ma i 1 service, 
the need to divert personnel from evaluative tasks to pol itical 
chores, constantly rotating staff, the reality of peak loads at 
certain times, etc., require that more man-days and resources be 
allocated than indicated in this analysis. This is especially 
true in the case of the institution liaison specialists and the 



parap rofess ional s , 
of the system, tho 
aud it 1 i a i son spec 
ma i n respon s i b i lit 
can be detailed to 



However, after the initia 
technical services liaison 
ialist will not be required 
ies. Hence, during pea!; pe 
assist the institution lia 



1 operational ization 
specialist and the 
full-time for their 
riods these specialists 
i son spec i a 1 i s t and 
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the administrative assistant. 

Overall, funding according to the preceding analysis is 
Relieved to provide a realisLic basis for launching the proposed 
evaluation system. 
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Recommendat i ons 



If DRDR adopts the evaluation system proposed in this report, the 
following recommendations are offered: 

— The DRDR eva 1 ua t i on-of f i ce should be staffed in 
accordance with the recommendations that appear in chapter six. 

-- The evaluation office, in conjunction with DRDR 
decision-makers, should prepare a M DRDR Annual Projection of 
Decisions and Associated Information Requirements. 11 This document 
provides the basic point of reference for all subsequent evaluation 
act i v i t i es . 



-- DRDR should 
system model described in 
-- DRDR should 
the procedures described 
-- DRDR shou 1 d 



train its staff to implement the general 
chapter two. 

standardize site visits according to 
in chapter five. 

immediately employ the sets of guidelines 
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that appear in appendix A. 
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-- A contract should be let with an external technical- 
services agency for the development of instrumentation and data 
anal ysi s. 

— A contract should be let for selecting and training 
expert judges. 

-- A contract should be let for developing and operating 
the proposed audit system. 

— DRDR should encourage the establishment of strong 
evaluation offices in the research and development institutions 
because those institutions provide the majority of basic data for 
the DRDR evaluation system. 

— DRDR should announce the intention and operational 
characteristics of the new evaluation system and seek to build 
credibility and acceptance for the system. 

— DRDR should appoint an advisory panel to assist with 
the development, evaluation, and implementation of the evaluation 
system. 

The team that prepared this report is willing to 
collaborate further with DRDR by providing assistance in the 
operationalization, implementation, and improvement of the proposed 
evaluation approach. The team agrees that the proposed evaluation 
approach can work. What remains now is for DRDR to choose an 
evaluation strategy, to relate it effectively to the new support 
policy, and to implement both. 
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Gu? del 1 nes for Appl i cants & Judges 
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Dec i s ion 1 



U. 5 „ Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 

Guidelines for Planning Grant Applications 

for 

New Research and Development ’nstitution in Education 

Applicants for planning grants must organize their 
proposals according to the following sequence. The instructions 
contain the criteria on which proposals will be judged; consequently, 
all information requested must be supplied. 

The various sections of the proposal will be evaluated 

according to the follov;ing system of weighting. 



Section WeTaht 



1 . 


New R & D Capabi 1 ity 


3 


1 1 . 


Mission or Problem Area 


3 


Ml. 


Work Plan 


4 


IV. 


Personnel 


4 


V. 


Geographic Location 


2 


VI . 


Relations with Other Institutions, 
Agencies, and Organizations 


1 


VIM 




2 


Financing 
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lc New R & D Capability 

A. Explain how the creation of the proposed institution will 
make a distinct or unique contribution to research and 
development capability in education -- a capability not 
al ready available In existing institutions -- or how it 
will significantly extend or improve wnat existing 
institutions can do. 

1 „ Describe the distinction in mission or problem area 
that will set the proposed institution apart in its 
substantive program. 

2. Describe any distinct role the proposed institution 
will play in its geographic area. 

B. Describe sources of funding besides the U. So Office of 
Education that might assist either in establishing the 
institution or funding specific programs once the 
institution is established. 

II. Mission or Problem Area 

A. Indicate the problems, needs, or opportunities that will 
guide the institution in setting its mission. 

B. Outline the problem area and explain the mission in terms 
specific enough to provide focus yet general enough to 
justify long-range programmatic work. 

C. Identify the target pupil populations, demographic areas, 
and economic settings of the proposed work. 

D. Indicate whether the outcomes of the program will be 
national as well as regional in eventual impact. 

E. Discuss the theoretical framework or rationale for the 
proposed work; or supply a review of relevant literature; 
or provide backgrouri data; or describe the discrepancy 
between current cok 1 tions and desired conditions; or cite 
special character i s t i cs of the geographic area; or otherwise 
support the view that the proposed mission needs to be 

pe rf o rmed . 

F. Discuss the extent to which the mission is in keeping with 
generally accepted educational values. 

G. Explain how research and development activities will 
satisfactorily deal with the problems, needs, or 
opportunities identified. 
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H. Relate the mission or problem area to priorities in 

educat i on . 

| Describe the relationship to educational priorities 
recently expressed by the President; the Department 
of Health, Education, and Welfare; the U. S. Office 
of Education; the National Canter for Educational 
Research and Development; or the Division of Research 
and Development Resources* 

2. Describe the relationship to educational priorities 
expressed by other leading individuals, organ i za t i ons , 
and agenc 1 es . 

3. In the absence of any demonstrable relationship to 
priorities expressed by government officials or other 
national figures, develop the argument that the 
selected problem area is nonetheless significant and 
merits support . 

4. Discuss the likelihood that the work will continue to 
be significant in the future, even if priorities shift 

5. Discuss the significance in social and economic terms, 
as well as from an educational point of view. 



111. Work Plan 

A. Specify in precise terms the scope of work to be performed. 

B. List the critical questions that must be answered to 
achieve the objectives of the planning grant. 

C. Identify the formal or informal planning techniques to be 
used . 

D. Supply a careful description of the mechanism to be used for 
specifying the mission of the institution within its 
chosen problem area. Indicate the capabilities of the 
people who will specify the mission and explain how they 
will make their final decision about the mission. 

E. Include a carefully worked out design for identifying, 
examining, rating, and choosing among alternative 
procedures for carrying out the mission of the institution. 
Provide sufficient information to assure that alternatives 
will be thoroughly considered and final choices soundly 
made . 

F. Delineate the procedures for using evaluation techniques to 
assess and to guide the planning process. 
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G. Explain how the planners will divide the planning tasks 

among themselves. 

1. Explain the method for allocating tasks. 

2. Demonstrate that a match exists between tasks assigned 
and competence to do them. 

3. Display a schedule for the use of the planning 
personnel showing their availability to carry out 
tasks in a planned series and to finish the work on 
time. 

H. Supply a complete schedule for the planning tasks. 

1. Supply starting points, milestone points, and ending 
points for major segments of the work. 

2. Justify the schedule, given the s cope- of - wor k statement. 

IV. Personnel 

A. Identify the person who will lead the planning effort. 

1. Describe his competency for this task. 

2. Indicate the amount of time he will devote to the 
work and the degree of res pons ibility he will accept. 

B. Describe the organizational structure and the work pattern 

to be used by the planning group. 

1. Explain the use of teams, committees, or task groups, 
indicating how they will be led and interrelated. 

2. Indicate the amount of time and degree of involvement 
for each member of the planning group. 

C. Characterize the personnel. 

1 . Supply data to show they have both competency and 
relevant experience. 

2. Discuss the kind and number of personnel needed to 
carry out each task. Demonstrate that a critical 
mass of talent is available. 

D. List the classifications of personnel who will assist with 

the planning. 

1. Show that contributions from persons in the substantive 
disciplines will be made wherever relevant. 
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2. Show that talent from technical fields such as 
planning, evaluation, and data processing is available. 

3. Show that sufficient specialty in both research and 
development Is present within the planning group. 

4. Show that the ultimate clients of the institution, 
such as school systems, are represented among the 
pi anne rs . 

5. Show that key audiences for the work of the institution, 
such as members of state legislatures or key public 
figures, are involved in the planning. 

V. Geographic Location 

A. Describe the geographic location of the proposed institution 
and its accessibility. 

B. Discuss what the location implies for the ability of the 
institution to cooperate with other organizations engaged 
in similar or related work. 

C. Discuss what the location implies for the ability of the 
institution to attract and retain capable staff members. 

VI. Relations with Other Institutions, Agencies, and 0 r gan i zat i ons 

A. Indicate what formal sponsorship or affiliation the 
proposed institution will have with existing institutions, 
agencies, or organizations. 

B. If the applicant has endoresments of the proposal from 
other institutions or groups, these may be displayed. 

Indicate whether the proposed work emerges out of work 
previously done in the endorsing institutions or groups. 

C. Explain what services will be required from other agencies. 
Give evidence to certify that they w" 1 1 be available. 

D. Describe, and if possible supply data on, the attitudes held 
by others who might ultimately impede the work of the 
proposed institution or interfere with its accomplishments, 

VII. Financing 

A. Supply a detailed budget for the planning effort. 

1. Specify the costs of personnel, equipment and materials, 
special services, travel, communications, and space 
rental - 
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2. Demonstrate that the proposed budget is sufficient to 
carry out the scope of work. 

3 . identify those points at which the budget might be 
reduced without impeding the planning. 

B- Describe the fiscal system to hold, disburse, and account 
for funds. 

K Discuss the practicality of the fiscal system from 
the point of view of the planners. 

2. Supply assurances and guarantees of fiscal 

responsibility on the part of the fiscal agent. 

C. Describe other resources that will support the planning 
effort. 
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Decision 1 



U . S. Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 



Guidelines for Judges 
of 

Planning Grant Applications 
f o r 

New Research and Development Institutions in Education 

Judges must assess proposals for planning grants 
according to their compliance with the criteria contained in 
the following set of questions. The questions are organized 
to parallel the sequencing the applicants followed in writing 
their proposals. All questions must be answered. 

The various s ions of the proposal will be 
evaluated according to the following system of weighting. 



S ect i on Weight 

1 . New R & D Capab i 1 i ty 3 

11. Mission or Problem Area 3 

111. Work Plan *+ 

1 V . Pe r sonne 1 ^ 

V. Geographic Location 2 

VI. Relations with Other 

Institutions, Agencies 1 

and Organizations 

V 1 1 . Financing 
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I. 



New R & D Capabilities 



1 10 



A. Will the creation of the proposed institution make a 
distinct or unique contribution to research and develop- 
ment capability in education--a capability not already 
available in existing i n st i tut i on s—or will it significantly 
extend or improve what existing institutions can do? 

1. Does any distinction in mission or problem aua 
set the proposed institution apart from others 
in its substantive program? 

2. Will the proposed institution play any distinct 
role in its geographic area? 

B. Are there sources of funding besides the U. S. 

Office of Education thct might assist either in 
establishing the institution or funding specific 
programs once the institution is established? 

11. Mission or Problem Area 

A. Do clearly identified problems, needs, or opportunities 
guide the institution in setting its mission? 

B. Is the problem area outlined and the mission explained 
in terms specific enough to provide focus yet general 
enough to justify long-range programmatic work? 

C. Are significant pupil populations, demographic areas, 
and economic settings singled out as targets for the 
proposed work? 

D. Will the outcomes of the work be national (as well as 
regional) in eventual impact? 

E. Is there a sound theoretical framework or rationale for 
the proposed work; or a persuasive review of relevant 
literature; or relevant and important background data; 
or a convincing accounting of a discrepancy between 
current conditions and desired conditions; or special 
characteristics of the geographic area that demand 
attention; or other evidence that the proposed mission 
should be performed? 

F. Is the mission in keeping with generally accepted 
educational values? 

G. Can research and development activities deal satisfactorily 
with the problems, needs, or opportunities identified? 
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H. Is the mission or problem area related to priorities 
in education? 

1. Is it related to educational priorities recently 
expressed by the President; the Department of Health, 
Education, and Welfare; the IL S. Office of Education; 
the National Center for Educational Research and 
Development; or the Division of Research and 
Development Resources? 

2. Is it related to educational priorities expressed 
by other leading individuals, organizations, and 
agenci es? 

3 # In the absence of any demonstrable relationship 
to priorities expressed by government officials 
or other national figures, is the selected problem 
area nonetheless significant enough to merit support? 

4 . Will the work continue to be significant in the 
future, even if national priorities shift? 

5. Will achieving the proposed mission be significant 
in social and economic terms, as well as from an 
educational point of view? 

III. Work Pi an 

A. Is the scope of work specified In precise terms? 

B. Are the questions to be answered critical to achieving 
the objectives of the planning grant? 

C. Will the proposed formal or informal planning techniques 
lead to successful completion of the scope of work? 

D. Will the mechanism proposed for specifying the mission of 
the institution within its chosen problem area lead to a 
focused statement of an important, authenticated mission? 

Are the people who will specify the mission competent to 
work within the proposed mechanism and to make the needed 
deci s i on? 

E # Is the design fo* identifying, examining, rating, and 

choosing among alternative procedures for carrying out the 
mission of the institution carefully worked out? Does it 
provide sufficient information to assure that alternatives 
will be thoroughly considered and final choices soundly made? 

F. Will the proposed evaluation techniques effectively assess 
and guide the planning process? 
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G. Will the best possible division of tasks among the 

planning personnel be effected? 

1. Will the method for allocating the tasks work in 
actual practice? 

2. WU 1 there be a close match between tasks assigned 
and competence to do them? 

3. Will the planning personnel conduct their tasks 
in a planned series and finish the work on time? 

H. Is the schedule for the planning tasks balanced and complete? 

1. Does it supply precise starting points* milestone 
points, and ending points for major segments of 
the work? 

2. Is the schedule closely related to the scope-of- 
work statement? 

IV. Personnel 



A. Will the person who will lead the planning effort be 
effect i ve? 



1. I s he competent for this task? 

2. Will he give enough time to, and take enough 
responsibility for, the work? 



B. Are the organizational structure and the work pattern to 
be used by the planning group appropriate for the task? 



1. Will the best possible use be made of the proposed 
work groups? 

2. Will the amount of time and degree of involvement for 
each member of the planning group be adequate to the 
task.' 



C. Are the personnel properly selected for the work? 

1. Will the personal competence and relevant experience 
of each lead to successful completion of the planning 
effort? 

2. Will a critical mass of specified talents be available? 

D. Will an adequate range of personnel ass? ’ with the plannin. 
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1. Will contributions from persons In the substantive 
disciplines be made wherever relevant? 

2 . Will talent from technical fields such as planning, 
evaluation, and data processing be available? 

3. Will sufficient specialty in both research and 
development be present within the planning group? 

4. Will the ultimate clients of the institution, 
such as school systems, be represented among the 
pi anner s? 

5. Will key audiences for the work of the institution, 
such as members of state legislatures or key public 
figures, be involved in the planning? 

V. Geographic Location 

A. Will the proposed institution be easily accessible to 
both clients and outsiders? 

B. Will the geographic location enable the institution to 
cooperate with other organizations engaged in similar or 
rel ated work? 

C. Will the location enable the institution to attract and 
retain capable staff members? 

VI. Relations with Other Institutions, Agencies, and Organizations 

A. Will formal sponsorship by or affiliation with existing 
institutions, agencies, or organizations contribute to 
the success of the proposed institution? 

B. Dc endorsements of the proposal from other institutions 
or groups contain promises of substantial assistance to 
the institution? Do endorsements come from people 
qualified to judge the proposed work? 

C. Will services required from other agencies be available? 

D. Given its selected mission and program, is the proposed 
institution likely to achieve a mutually supportive 
relationship with outside agencies and groups? 

VII. Fi nanci ng 

A. Is the budget for the planning effort carefully prepared? 
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1 . Is it complete? 

2. Is it suf f I cl ent? 

3* Is it limited to what I s recju i red? 

b - Is the fiscal system complete and carefully devised? 

1 . Is it easy to jse? 

2. Are adequate assurances and guarantees of fiscal 
responsibility on the part of the fiscal agent 
suppl i ed? 

C. Are other resources available to support the plannin 
effort? 
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Dec i s j on 2 



U . S „ Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 



Guidelines for New Institution Applications 

for 

Recipients of Planning Grants 
for 

New Research and Development Institutions in Education 



Recipients c planning grants must perform the 
following tasks in preparing a proposal for the creation of a 
new research and development institution. These instructions 
contain the criteria on which proposals will be judged; consequently 
all information requested must be supplied. 

The various sections of the grants will be evaluated 
according to the following system of weights; 

Sect i on 



|. Mission or Problem Area 
1 | . Management 
Ml. Pe r sonne 1 

IV. Planning and Evaluation System 
V. Program Coordination and Balance 
VI. Research Activities 

Deve 1 opment Activities 



V 1 I 
VM I 



Relations with Other Institutions, 
Agencies, and Organizations 



We i ght 
4 
4 
4 

3 

1 

2 

2 

1 



IX, 



Schedu 1 e 
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1 



1 16 



X. Faci 1 i ties 
XI. Fi nartc i ng 
XII. Communication 



1 





117 



Summary of Selected Information Contained in 
Original Application for Planning Grant 



The following information may be drawn from the original planning 
grant proposal and summarized briefly; or as an alternative, 
the proposal may be attached and the location of the following 
information indicated. In either case, any changes made since 
the original application was filed must be noted. 



1. Explain how the creation of the proposed institution will 
make a distinc. or unique contribution to research and 
development capability in education — a capability not already 
available in existing institutions — or how it will significantly 
extend or improve what existing institutions can do. 

a. Describe the distinction in mission problem area 
that will set the proposed institution apart in its 
subs tan t i ve program. 

b. Describe any distinct role the proposed institution will 
play in its geographic area. 

2. Describe funding sources in addition to the U.S, Office 

of Education that might be able to assist either in establishing 
the institution or funding specific programs once the 
institution is established. 

3. Justify the assertior that the problems, needs, or 
opportunities identified can be dealt with satisfactorily 
through research and development activities and cannot 

be better approached by other means. 

4. Describe the relationship between the proposed work and 
what has been accomplished previously or is currently under 
way elsewhere. 

5. Explain why the mission or programs of the institution 
require long-range programmatic research and development in 
an institutional setting and cannot be handled 
satisfactorily through the support of single projects. 

6. Describe the procedures used for identifying, examining, 
rating, and choosing among alternative procedures 



that might have been used to carry out the mission of the 
institution. Include sufficient information to give 
assurance that the consideration of alternatives was 
thorough and that the final choices were sound. 

7. Describe the geographic location of the preposed institution 
and its accessibility. 

a. Discuss what the location implies for the ability of 
the institution to cooperate with other organizations 
engaged in similar or related work. 

b. Discuss what the location implies for the ability of 
the institution to attract and retain capable staff 
members . 

8. Indicate what formal sponsorship or affiliation the pro- 
posed institution will have w th existing institutions, 
agencies, or o rgan i zat i ons . 

9- Descr I be, and , i r possible, supply data on the attitudes 
toward the proposed institution held by others who might 
ultimately impede its work or interfere with its 
accomp 1 i shments . 



f4ew Tasks to Be Performed 



Although several of the following tasks may have been 
performed in writing the original application for a planning grant, 
a more elaborate and careful restatement of them is expected from 
actual recipients of planning grants. Quoting the original 
application is not sufficient for these tasks. Inasmuch as the 
instructions below contain the criteria on which proposals will 
be judged, the steps must be carried out as described in this 
work statement. 

1. Mission or Problem Area 

A. Clearly specify the problems, needs, or opportunities 
that will guide the institution in setting its mission. 

B. Delineate the problem area and explain the mission in 
terms specific enough to provide focus vet general enough 
to justify long-range programmatic work. 
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C. Describe the pupil populations, demographic areas, and 
economic settings that are the targets of the proposed 
work. 

D. Authenticate the problems, needs, or opportunities by 
presenting a theoretical framework or rationale for the 
proposed work; or supply a thorough review of relevant 
literature; or delineate the discrepancy between current 
conditions and desired conditions; or cite special 
characteristics of the geographic area; or otherwise establish 
conclusively that the proposed mission needs to be performed. 

E. Present data to demonstrate that the mission for the 
proposed Institution is necessary and significant. 

F. Discuss the extent to which the mission is in keeping 
with generally accepted educational values. 

G. Indicate whether the outcomes of the program will be 
national as well as regional in eventual impact. 

H. Demonstrate that the objectives specified for the 
institution, when taken as a whole, provide a coherent 
basis for the establishment of program plans. 

I. Justify the assertion that research and development 
activities will satisfactorily deal with the problems, 
needs, or opportunities identified. 

J. Relate the mission or problem area to priorities in 
education. 

1. Describe the relationship to educational priorities 
recently expressed by the President; the Department 
of Health, Education, and Welfare; the U. S. Office 
of Education; the National Center for Educational 
Research and Development; or the Division of 
Research and Development Resources. 

2. Describe the relationship to educational priorities 
expressed by other leading individuals, organiza- 
tions, and agencies. 

3. In the absence of uny demonstable relationship to 

priorities exp res u y government officials or 

other national f us, develop the argument that 

the selected problem area is nonetheless significant 
and merits support. 
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4. Discuss the likelihood that the work will continue 
to be significant In the future, even if priorities 
shift. 

5. Discuss the significance in social and economic 
terns, as well as from an educational point of view. 

i I . Management 

A. Supply evidence that the required institutional arrangements 
can be consummated; e. g. , that the proposed research and 
development enterprise can achieve incorporation or that 

it can receive support and cooperation from the host institution. 

B. Outline the governing structure of the new institution, 
explaining the part staff members as well as any governing 
board or external advisory committees will play. 

C. Describe how decision-making authority and responsibility will 
be allocated within the institution. 

D. Present the institution's mechanism for refining its 
mission as it gains experience. 

E. Attach copies of personnel pol i ci es developed for the 
i n st I tut i on . 

F. Describe the budgeting process established for the institution. 

Ill, Pe rsonnel 

A. Specify the roles of staff members at all levels, giving 
brief job descriptions for major positions. 

B. Explain how staff qualifications are relevant to conducting 
the proposed mission and programs. Demonstrate how staff 
experiences are sufficient in variety and comprehensiveness. 

C. List the names and provide biographical data for the key 
personnel who will work in the proposed institution when it 
i s funded . 

D. Supply evidence that persons engaged in the planning who 
will continue as permanent staff members, understand the plan, 
agree with it, and are committed to carrying it out. 

E. Describe the sources that will provide a continuing supply of 
regular personnel as well as consultants. 

IV. Planning and Evaluation System 

A. Project the means to assess and modify institutional goals on 
a continual basis as the programs begin to have impact on 
the problem basi s. 
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B. Project procedures to identify and assess alternative 
strategies for achieving the goals. 

C. Project procedures to monitor and assess activities at the 
o per at I onal level. 

D. Project methods to determine the extent to which Institutional 
objectives are being achieved. 

E. Project means to draw expertise from a variety of disciplines 
as needed in developing program plans. 

F. Project procedures to relate budgets to programs. 

V. Program Coordination and Balance 

A. Identify the method to be used to coordinate and interrelate 
programs so that they are mutually reinforcing; if possible, 
provide for cumulative results. 

B. Develop a plan that demonstrates a reasonable balance in 
allocating resources between activities that accomplish 
organi zat i onal 1 y defined objectives and exploratory projects 
that involve high risk but promise high payoff. 

C. Demonstrate that the proposed balance between research and 
development is in keeping with the mission of the institution 
and the state of knowledge in the field. 

VI, Research Activities 

A. Specify research objectives. 

B. Demonstrate that the research objectives are relevant both 
to the development objectives and to the overall goals of 
the institution. 

C. Describe proposed research designs, giving evidence of their 
scientific adequacy. 

D. Explain how the institution will draw information both from 
related research efforts and from the existing literature. 

VII. Development Activities 

A. Specify development objectives. 

B. Demonstrate that these objectives are relevant both to research 
objectives and to overall goals. 

C. Set forth a development process and justify its choice. 
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D. Outline the procedures and the timetable to accomplish 
objectives. 

E. Indicate how unavailable, specialized development capabilities 
will be acquired, e. g. , through consulting arrangements, 
subcontracts, or other means. 

Vjl I - Relations with Other Institutions, Agencies, and Organizations 

A. Indicate what arrangements will be made to obtain formal 
sponsorship from or affiliation with existing institutions, 
agencies, or organizations. 

B. Describe any collaborative relationships with outside 
organizations that will enhance the work of the proposed 
institution. 

C. Describe the assistance available from existing institutions 
that will help start the new enterprise. 

D. Explain what services other agencies will need to provide. 

Give evidence to certify their availability. 

IX # Schedule 

A. Specify starting points, milestone points, and ending points 
for all major activities to be undertaken in creating the 
institutional framework, including the recruitment of staff 
and acquisition of physical facilities. 

B. Specify the starting points, milestone points, and ending 
points for all programs and all program components. 

X. Faci 1 i t i es 

A. Describe the housing available for the Institution while It 
is getting started. 

B. Describe the facilities being suggested for long-term housing 
of the institution. 

C. Describe the criteria and procedures to be used to select 
a building site if construction is proposed. 

XI. Fi nanci ng 

A. Supply a detailed budget for the institution. 

1. Specify the costs of personnel, equipment and materials, 
special services, travel, communications, and space 
rental . 
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2. Demonstrate that the proposed budget is directly related 
to, as well as sufficient to carry out, the proposed work. 

3. Identify those points in the budget that might be reduced 
without impeding progress of the program, 

B. Describe the fiscal system to hold, disburse, and account 
for funds. 

C. Describe other sources of support for the planning effort. 

XII* Commun i cat i on s 

A. Describe a plan for publicizing the institution that will 

make the nature of its work clear, demonstrate its significance, 
and arouse interest in its potential. 
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Dec i s i on 2 



U.S. Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 



Guidelines for Judges 
of 

New Institution Applications 
for 

New Research and Development Institutions in Education 



O 

ERIC 



Judges must assess the work statements of recipients of 
planning grants according to their standing on the criteria 
contained in the following set of questions. The questions are 
organized so that they parallel the sequencing the recipients 
were required to follow in writing their work statements. All 
questions must be answered. 

The various sections of the statement 
according to the following system of weights: 

Section 



w i 



be evaluated 



I. Mission or Problem Area 
I I . Management 
Mi. Per sonne 1 

JV. Planning and Evaluation System 

V. Program Coordination and Balance 

VI. Research Activities 

VII. Development Activities 

VIII. Relations with Other Institutions, 
Agencies, and Organ i zat i ons 

IX. Schedule 

124 
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k 

4 

4 

3 

1 

2 

2 

1 

3 
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X. 


Fac i 1 i 1 1 es 


1 


XI . 


F 1 nanc \ ng 


2 


XII. 


Commun i cat ion 


1 




132 



126 



Summary of Selected Information Contained in 
Qrioinal Application for Planning Grant 




Recipients of planning grants were permitted to draw from 
the original planning grant proposal and summarize it briefly; as an 
alternative they could attach the proposal and indicate the location of 
the following information. in either case, recipients were to point 
out any changes made since the original application was filed. 

1. Will the creation of the proposed institution make a distinct 
or unique contribution to research and development capability 
in education -- a capability not already available in existing 
institutions — or w« 1 1 it significantly extend or improve what 
existing institutions can do? 

a. Does any distinction in mission or problem area set the 
proposed institution apart from others in its substantive 
program? 

b. Will the proposed institution play any distinct role in its 
geographic area? 

2. Are there sources of funding in addition to the U.S. Office of 
Education that might assist either in establishing the 
institution or funding specific programs once' the institution 

i s establ i shed? 

3. Can research and development activities deal satisfactorily with 
the problems, needs, or opportunities identified? 

4. Does the relationship between the proposed work and what has 
been accomplished previously or is under way elsewhere 
justify founding a new institution? 

5. Does the proposed mission or programs require long-range 
programmatic research and development in an institutional 
Setting rather than through the support of single projects? 

6. Is the design for identifying, examining, rating, and choosing 
among alternative procedures for carrying out the mission of the 
institution carefully worked out? Does it provide information 
sufficient to assure that alternatives will be thoroughly 
considered and final choices soundly made? 

7. Will the proposed institution be easily accessible to both 
clients and outsiders? 

a. Will the geographic location enable the institution to 
cooperate with other organizations engaged in similar or 
related work? 
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b. Will the location enable the institution to attract 
and retain capable staff members? 

8. Will formal soonsorhip by or affiliation with existing 
institutions, agencies, or organizations contribute to the 
success of the proposed institution? 

9. Given its selected mission and program, is the propos d 
institution likely to achieve a mutually supportive relationship 
with outside agencies and groups? 



New Tasks to be Performed 

Although several of the following tasks may have been 
stated in the original application for a planning grant, a more 
elaborate and careful listing of them is expected from actual 
recipients. Quoting the original proposal is not sufficient for 
these tasks. The questions below contain the criteria on which 
proposals will be judged. All questions should be answered. 



1. Mission or Problem Area 

A. Do clearly identified problems, needs, or opportunities 
guide the institution in setting its mission? 

B. Is the problem area outlined and the mission explained in 
terms specific enough to provide focus yet general enough 
to justify long-range programmatic work? 

C. Are significant pupil populations, demographic areas, and 
economic settings singled out as targets for the proposed 
work? 

D. Is there a sound, theoretical framework or rationale for 
the proposed work; or a persuasive review of relevant 
literature; or relevant and important background data; or 
a convincing accounting of a discrepancy between current 
conditions and desired conditions; or special characteristics 
of the geographic area that demand attention; or other 
evidence that the proposed mission should be performed? 

£. Is the mission for the proposed institution necessary and 
significant? 

F. Is the mission in keeping with generally accepted 
educational values? 
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G. Will the outcomes of the work be national as well 
as regional In Impact? 

H. Do the objectives soecifieci for the institution* when 
taken as a whole* provide a coherent basis for the 
establishment of program plans? 

I. Can research and development activities deal sat i sf acto r i 1 y 
with the problems, needs, or opportunities identified? 

J. Is the mission or oroblem area related to priorities in 
educat i on? 

1. Is it related to educational priorities recently 
expressed by the Pres ent; the Department of Health, 
Education, and Welfare; the U. S. Office of Education; 
the National Center for Educational Research and 
Development; or the Division of Research and 
Development Resources? 

2. Is it related to educational priorities expressed by 
other leading individuals, o rgan i za t i ons , and agencies? 

3* In the absence of any demonstrable relationship to 

priorities expressed b> government officials or other 
national figures, does it develop the argument that 
the selected problem area is nonetheless significant 
and merits support? 

4. Will the work continue to be significant in the future, 
even if priorities shift? 

5« Will achieving the proposed mission be significant in 
social and economic terms, as well as from an 
educational point of view? 

I I , Management 

A. Is there adequate evidence that the required institutional 
arrangements can be consummated, e.g., that the proposed 
research and development enterprise can achieve 

i ncorporat i on or that it can receive support and 
cooperation from the host institution? 

B. Will the governing structure of the new institution, 
including staff members as well as any governing board 
or external advisory committees, lead to effective and 
productive policy decisions? 

C. Will the authority and responsibility allocated within 

the institution lead to timely and effective adm i n i s t ra t i ve 
dec i s i ons? 
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D. Will the institution introduce needed refinements into 
its mission as it gains experience? 

E Will the personnel policies developed for the institution 
contribute effectively to achieving the selected r & d 
mi ss i on? 





F. 


Is the budgeting process established for the institution 
adequate? 


1 1 . 


Re rsonne 1 




A. 


Is the Staffing plan adequate? 




B. 


Will staff qualifications match the needs of the 
institutional mission and proposed programs? 




C. 


Does previous experience of the key personnel involved 
lead to confidence that the institution will succeed? 




D. 


Do the planner and future staff members understand the 
plan? Are they in agreement wi th it and committed to 
car ry i ng it out? 




E. 


Is there an adequate pool of manpower for regular personnel 
appointments as well as for consultants? 


IV. 


Planning and Evaluation System 




A. 


Will the institution assess and modify its goals 'on a 
continual basis as its programs begin to have impact on 
the problem area? 




B. 


Will alternative strategies for achieving the goals be 
identified and assessed? 




C. 


Will activities at the operational level be monitored and 
assessed? 




D. 


Will the institution determine the extent to which its 
objectives are being achieved? 




E. 


Will expertise from a variety of relevant disciplines be 
employed as needed in developing program plans? 




F. 


Will budgets and programs be related to each other 
effect ively? 


V. 


Program Coordination and Balance 




A. 


Will program activities within the institution be 
mutually reinforcing and cumulative? 



O 
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B. Will there be a reasonable balance between activities 
that accomplish o rgan i za t i ona 1 1 y defined objectives and 
those addressed to high-potential exploratory projects? 

C. Will the balance between research and development be in 
keeping with the mission of the institution and the state 
of knowledge in the field? 

VI. Research Activities 

A. Are research objectives clearly specified? 

B. Are the research objectives relevant both to the 
development objectives and to the overall goals of the 
i ns t i tu t i on? 

C. Are the proposed research designs scientifically 
adequa te ? 

D_ Will the institution keep itself apprised of past and 
current research? 

V I I . Development Activities 

A. Are development objectives clearly specified? 

B. Are development objectives relevant both to research 
objectives and to the overall goals of the institution? 

C. Is the development process adequate? 

D. Are the procedures and the timetable for achieving the 
obj ec t i ves realistic? 

E. Will necessary specialized development capabilities not 
available within the institution be acquired? 

Vlll. Relations with Other Institutions, Agencies, and Organizations 

A. Will sponsorship by or affiliation with existing 
institutions contribute to the success of the institution? 

B. Will any collaborative relationships with outside 

organ i za t ions enhance the work of the proposed institution? 

C. Will assistance be available from existing institutions 
to help start the new enterprise? 

D. Will services requ i red from other agencies be provided? 
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IX. Schedule 

A. Is the timetable reasonable for creating the institutional 
framework, including the recruitment of staff and 
acquisition of physical facilities? 

B. Is the timetable reasonable for carrying out the program? 

X. Facilities 

A. Will there be adequate housing available while the 
institution is getting started? 

B. Will the facilities being suggested for proposed long-term 
housing of the institution be suitable? 

C. Will acceptable criteria and procedures be used to select 
a building site if construction is proposed? 

XI. Financing 

A. Is the budget for the institution carefully prepared? 

1 . Is it comp le te? 

2. Is it directly related to the proposed work as well 
as sufficient? 

3. Is it limited to what is required? 

B. Is the f complete and carefully devised? 

C. Are ot! ’’ u resources available to upport the 

p 1 ann i ng efrort? 

XI 1. Communications 

A. Will the clients and audiences of the institution 

clearly understand the nature and significance of its 
work and be Interested in its potential? 
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Decision 3 



U. S. Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 



Guidelines for Program Plans 
for 

Developing Research and Development Institutions 
Receiving Institutional Support from U. S. Office of Education 



Research and development institutions preparing program 
plans for support by the U. S. Office of Education must follow 
the instructions below. They contain the criteria on which program 
plans will be judged; thus all information requested must be supplied. 

The various sections of the proposal will be evaluated 
according to the following system of weighting. 



Sect i on We i gh t 

I. Institutional Mission, Capability, 

and Sett i ng A 

I 1 . Program A 

ill. Management k 

IV. Per sonne 1 ^4 

V» Program Coordination and Balance 2 

VI. Research Activities 3 

VII. Development Activities 3 

VI I L Relations with Other Institutions, 

Agencies, and 0 rgan i za t i ons 2 

IX. Schedule 3 

X . Fac i 1 i t i es 1 

X l „ F i nanc i ng 2 

XII. Commun i cat i on 139 i 
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Summary of Selected Information Contained in Proposal for New 
Research and Development Institution 

The following information may be drawn from the original 
proposal; as an alternative, the proposal may be attached and the 
location of the following information indicated. In either case, 
any changes made since the original application was filed must be 
indicated. It is expected that the Basic Program Plan will contain 
more elaborate information. For example, it is expected that the 
delineation of the problem arer and mission of the institution will 
have been sharpened and clarified considerably during the writing 
of the Basic Program Plan. 

I. Institutional Mission, Capability, and Setting 

A. Explain how the creation of the proposed institution 
will make a distinct or unique contribution to research 
and development capability in education -- a capability 
not already available in existing institutions -- or how 
it will significantly extend or improve what existing 
institutions can do. 

1. Describe the distinction in mission or problem 
area that will set the proposed institution apart 
in its substantive program. 

2. Describe any distinct role the proposed institution 
will play in its geographic area. 

B. Describe sources of funding besides the U. S. Office of 
Education that might assist either in establishing the 
institution or funding specific programs once the 
institution is established. 

C. Outline the problem area and explain the mission >, . ~ rms 

specific enough to provide focus yet general enough to 
justify long-range programmatic work. 

D. Identify the pupil populations, demographic areas, and 
economic settings that are the targets of the proposed 
work. 

E. Relate the mission or problem area to priorities in 
educa t i on . 
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1. Describe the relationship to educational priorities 
recently expressed by the President; the Department 
of Health, Education, and Welfare; the U. S. Office 
of Education; the National Center for Educational 
Research and Development; or the Division of Research 
and Development Resources. 

2. Describe the relationship to educational prior ; ties 
expressed by other leading individuals, organizations, 
and agencies. 

3. In the absence of any demonstrable relationship to 
priorities expressed by government officials or other 
national figures, develop the argument that the 
selected problem area is nonetheless significant and 
merits support . 

4. Discuss the likeli hood that the work will con t i nue to 
be significant in the future, even if priorities 
shift. 

5. Discuss the significance in social and economic terms, 
as well as from an education point of view. Indicate 
whether the social and economy contributions are 
expected to be regional or nr onal in eventual 

i mpact . 

F. Justify the assertion that the p oblems, needs, or 
opportunities identified can be dealt with sa t i sf ac tor i 1 y 
through research and development activities and cannot be 
approached better by another means. 

G. Describe the relationship between the proposed work and 
what has been accomplished previously or is under way 

e 1 sewhere. 

H. Explain why the mission or programs of the institution 
require long-range programmatic research and development 
in an institutional setting and cannot be handled 

sa t i sf actor i 1 y through the support of single projects. 

I. Describe the procedures used for identifying, examining, 
rating, and choosing among alternative procedures that 
might have been used to carry out the mission of the 
institution. Include sufficient information to give 
assurance that the consideration of alternatives was 
thorough and that the final choices were sound. 

J. Describe the geographic location of the proposed 
institution and its accessibility. 
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1. Discuss what the location implies for the ability of 
the institution to cooperate with other organizat ions 
engaged in similar or related work. 

2. Discuss what the location implies for the ability of 
the institution to attract and retain capable staff 
members . 

K. Indicate what formal sponsorship or affiliation the 

p roposed institution will have with existing institutions, 
agencies, or organ i zat i ons . 

L- Describe, and if possible supply data on, the attitudes 
toward the proposed institution held by others who might 
ultimately impede its work or interfere with its 
accompl ishments. 

Additional Tasks to Be Performed 

Although several of the fol lowing tasks may have been 
pe rf o rmed in writing the prop os al for establishing the institution, 
more elaborate information is needed for the program plans. Quoting 
the original proposal is insufficient. Inasmuch as the instructions 
below contain the criteria on which proposals will be judged, all 
taks indicated must be carried out in writing the program plans. 

I I . Program 

A. Present the configuration of mi ss i on/program/ component/ 
products so that their completeness and their internal 
consistency will be apparent, 

1. Demonstrate that the products of the program are 
necessary for the institution to achieve its mission. 

2. Demonstrate that programs are derived from the mission, 
that components constitute coherent programs, and that 
components will produce the anticipated products. 

B. Specify defensible, clear, important objectives for the 
p rog ram. 

1. Supply evidence to justify selection of the 

objectives and to demonstrate that they were chosen 
reasonably from among a set of alternatives. 
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2. State the objectives in operational terms, clearly 
specifying what the terminal product of each objective 
will be. For example, is the product to be new 
knowledge, some new form of educational practice, or 
new instructional materials? 

3- Indicate the pupil populations, demographic areas, 
and economic settings to be affected by achievement 
of the objectives. 

4. Demonstrate that the objectives strike a favorable 
balance between addressing critical needs and using 
available opportunities for meeting needs. 

5- State whether achievement of the objectives is 

expected to make an incremental improvement, a sharp 
modification, or a complete change in the schools. 

C. Specify defensible, clear, and relevant procedures for 

achieving the objectives. 

1. Supply evidence to justify the selection of the 

procedures and to demonstrate that the choice among 
the available alterna yes was reasonable. 

2 ♦ State the procedures clearly in operational terms. 

3. Show how the procedures are related to achieving the 
ob j ec t i ves . 

k. Describe the mechanisms for evaluating the program 
processes and the products, and demonstrate their 
adequacy for self-correction and redirection. 

III. Management 

A. Supply evidence that the management has established and 

can make effective use of procedures for: 

l. Planning and evaluation 

2. Recruiting, assigning, training, evaluating, promoting, 
and compensating personnel 

3- Monitoring detailed operations within all programs 

4. Arranging for outside consulting and subcontract 
se rv i ces 

5. Performing housekeeping functions 
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IV. 



V. 



VI . 



6. Operating a fiscal system with adequate safeguards, 
records, and reports. 

B. Supply evidence that personnel who have guided and 
directed the development of the program plans can 
provide effective leadership in carrying them out. 

C. Provide an administrative organization chart for the 
institution and describe its relationship to the 
organization of the basic program. 

Pe rsonne 1 

A. Specify the roles of staff members at all levels, giving 
brief job descriptions for major positions. 

B. Explain how staff qualifications are relevant to carrying 
out the proposed program. Demonstrate how staff 
experiences are sufficiently varied and comprehensive. 

C. List the names and provide biographical data for the key 
staff personnel . 

D. Supply evidence that persons who engaged in the planning 
and will continue as permanent staff members understand 
the plan, are in agreement with it, and are committed to 
ca r ry i ng it ou t . 

E. Describe the sources that will provide a continuing 
supply of regular personnel as well as consultants. 

Program Coordination and Balance 

A. Demonstrate that the proposed programs are mutually 

reinforcing and explain the degree to which they will be 
cumulative in their effect. 

B _ Demonstrate that there is a reasonable balance between 
activities that accomplish organ i zat i ona 1 1 y defined 
objectives and exploratory projects that involve high risk 
but promise high payoff. 

C- Demonstrate that the proposed balance between research 
and development is in keeping with the mission of the 
institution and the State of knowledge in the field. 

Rt ' larch Activities 

A. Specify research objectives. 
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B. Demons t ra te that the research objectives are relevant 
both to the development objectives and to the overall 
goals of the institution. 

C. Describe proposed research designs, giving evidence of 
their scientific adequacy. 

D . Exp 1 a i n how the i ns t i tu t t on will d raw i n f o rr>ia t i on bo t h 
from related research efforts and from the existing 
1 i terature. 

VII. Development Activities 

A. Specify development objectives. 

B. Demonstrate that these objectives are relevant both to 
research objectives and to overall goals. 

C. Set forth a development process and justify its choice. 

D. Outl i ne the procedures and the timetable to accompl i sh 
the ob j ect i ves . 

E. Indicate how specialized development capab i 1 i t i es not 
available within the institution will be acquired, e.g., 
through consulting arrangements or subcontracts. 

VIM. Relations with Other Institutions, Agencies, and Organizations 

A. Describe any co 1 I abo ra t i ve relationships with schools, 
colleges, state departments of education, research and 
development organizations, E5EA Title I and Title Ml 
projects, or other federal programs-. Explain how these 
relationships will assist the institution in achieving 
i ts obj ect i ves , 

B • Explain wha t services other a gene i es will need to provide. 
Give evidence to certify their ava i 1 ab i 1 i ty . 

IX. Schedule 

A. Specify the starting points, milestone points, and ending 
po ints for oil progr ams and all prog ram componen t s . 

B. Identify the points at which recycling of activity can 
correct unsatisfactory outcomes. 

C. Identify the possibilities for accelerating or decelerating 
the tempo in case of changes in personnel or financial 
support . 
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Facilities 

A. Describe the equipment, materials, and physical 
facilities to be used in carrying out the program. 

B. Demonstrate that these are appropriate, sufficient, 
and economical for the work. 

F i nanc i ng 

A. Supply a detailed budget. 

1. Specify the costs of personnel, equipment and 
materials, special services, travel, communications, 
and space rental . 

2. Demonstrate that the proposed budget is related to 
the proposed work and is sufficient to carry it out. 

3. Identify those points in the budget that might be 
reduced without impeding progress of the program. 

B. Describe the fiscal system to hold, disburse, and 
account for funds. 

C. Describe other resources available to support the planning effort 
Commun i ca t i ons 

A. Describe a plan for publicizing the institution that will 
make the nature of its work clear, demonstrate its 
significance, and arouse interest in its potential. 

B. Describe how this can be done so that it will increase the 
constituency for research and development activities in 
educa t i on . 
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Dec i s i on 3 



U. S- Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 




Guidelines for Judges 
of 

Program Plans 
for 

Developing Research and Development Institutions 
Receiving Institutional Support from U. S. Office of Education 

Judges must assess the program plans according to their 
standing on the criteria contained in the following questions. The 
questions are organized so that they parallel the sequencing the 
planners followed in writing their program plans. All questions 
must be answered. 

The various sections of the proposal will be evaluated 
according to the following system of weighting. 



Sect i on Weight 

I. Institutional Mission, Capability, 

and Setting 4 

I | . Program 4 

III. Management 4 

IV. Personnel 4 

V. Program Coordination and Balance 2 

VI. Research Activities 3 

VII. Development Activities 3 

VIM . Relations with Other Institutions, 

Agencies, and Q r gan i za t i ons 2 

I X . Schedu 1 e 3 

X. Fac i 1 i t ies 1 

XI. F i nanc i ng 2 

XII. Communication 147 i 
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Summary of Selected Information Contained in Proposal for New 
Research and Development Institution 

Planners were permitted to draw from the original proposal, 
or to attach the proposal and indicate the location of the following 
information. In either case, planners were to point out any changes 
made since the original proposal was filed. 

1. Institutional Mission, Capability, and Setting 

A. Will the creation of the proposed institution make a 
distinct or unique contribution to research and 
deve 1 opment ca pa b i 1 i ty in education — a capability not 
a 1 ready avai lable in ex isting institutions or will it 
significantly extend or improve what existing 
institutions can do? 

1. Does any distinction in mission or problem area 
set the proposed institution apart from others in 
its substantive program? 

2* V/ i 1 1 the proposed institution play any distinct 
role in its geographic area? 

Be Are there sources of funding besides the U. S. Office 
of Education that might assist either in establishing 
the institution or funding specific programs once the 
institution is established? 

C. Is the problem area outlined and the mission explained 
in terms specific enough to provide focus yet general 
enough to justify long-range programmatic work? 

D. Are significant pupil populations, demographic areas, 
and economic settings singled out as targets for the 
proposed work? 

E. Is the mission or problem area related to priorities 
i n educa t i on? 

1. Is it related to educational priorities recently 
expressed by the President; the Department of 
Health, Education, and Welfare; the U. S. Office 
of Education; the National Center for Educational 
Research and Development; or the Division of 
Research and Development Resources? 

2. Is it related to educational priorities expressed 
by other leading individuals, o rgan i zat i ons , and 
agenc i es? 
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3. In the absence of any demonstrable relationship 
to priorities expressed by government officials 
or ether national figures, does *t develop the 
argument that the selected problem area is 
nonetheless significant and merits support? 

4. Will the work continue to be significant even if 
priorities shift? 

5. Will achieving the proposed mission be significant 
in social and economic terms, as well as from an 
educational point of view? 

F. Are research and development activities the best means 
of dealing with the problems, needs, or opportunities 

i dent i f i ed? 

G. Does the relationship between the proposed work and 
what has been accomplished previously or is under way 
elsewhere justify supporting this program? 

H. Do the proposed mission or programs require long-range 
programmatic research and development in an 
institutional setting rather than through the support 
of single projects? 

I. Is the design for identifying, examining, rating, and 
choosing among alternative procedures for carrying out 
the mission of the institution carefully worked out? 
Does it provide information sufficient to assure that 
alternatives will be thoroughly considered and final 
choices soundly made? 

J. Will the proposed institution be easily accessible to 
both clients and outsiders? 

1. Will the geographic location enable the institution 
to cooperate with other organ i za t i ons engaged in 
similar or related work? 

2. Will the location enable the institution to attract 
and retain capable staff members? 

K. Will formal sponsorship by or affiliation with existing 
institutions, agencies, or organ i za t i ons contribute to 
the success of the proposed institution? 

L. Given its selected mission and program, is the proposed 
institution likely to achieve a mutually supportive 
relationship with outside agencies? 
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Additional Tasks to Be Performed 
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Although several of the following tasks may have been 
performed in writing the proposal for establishing the institution, 
more elaborate information is needed for the program plans. Quoting 
the original proposal is not sufficient. The questions below 
contain the criteria on which proposals will be judged. All 
questions must be answered. 

I I . Program 

A. Is the configuration of mission/program/components/ 

products both complete and internally consistent? 

1. Will the institution advance its mission if it 
completes the projected products? 

2. Does the proposed configuration of parts constitute 
a consistent set of linked elements? 

B. Are the objectives defensible, clear, and important? 

1. Are the selected objectives the best choice from 
among the range of possible objectives? 

2. Is the terminal product of each objective clearly 
spec i f i ed? 

3. Are the targets of the program --pupil populations, 
demographic areas, and economic settings -- clearly 
spec i f i ed? 

4. Do the objectives strike a favorable balance 
between addressing critical needs and using 
available opportunities for meeting needs? 

5. Is the expected outcome -- either an incremental 
improvement, a sharp modification, or a complete 
change in the schools -- the best choice of the 
three a 1 te rna t i ves, g i ven the nature of the problem? 

C. Are the procedures for achieving the objectives 

defensible, clear, and relevant? 

1. Are the selected procedures the best choice from among 
possible alternatives? 

2. Are the procedures clearly stated in operational 
te rms? 
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3. Are the procedures clearly related to achieving the 
obj ect i ves? 

4. Will the selected evaluating mechanisms lead to 
effective se 1 f- cor rect i on and redirection of both 
the program processes and the products? 

lii. Management 

A. Will management make effective use of procedures for: 

1. Planning and evaluation? 

2. Recruiting, assigning, training, evaluating, 
promoting, and compensating personnel? 

3. Monitoring detailed operations within all programs? 

4. Arranging for outside consulting and subcontract 
se rv i ces? 

5. Performing housekeeping functions? 

6. Operating a fiscal system with adequate safeguards, 
records, and reports? 

B. Will the personnel who have guided and directed the 
development of the program plans provide effective 
leadership in carrying them out? 

C. Is the administrative organization of the institution 
related effectively to the organization of the program? 

IV, Personnel 

A. Is the staffing plan adequate? 

B. Do the qualifications of the staff match the needs of 
the program? 

C. Does previous experience of the key personnel who have 
agreed to work when the program is funded lead to 
confidence that it will succeed? 

D. Do the persons engaged in the planning who will continue 
as permanent staff members understand the plan? Are 
they in agreement with it and committed to carrying it 
out? 

E. Is there an adequate pool of manpower for regular personnel 
appointments as well as for consultants? 
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V. Program Coordination and Balance 

A. Will program activities within the institution be 
mutually reinforcing? 

B. Will there be a reasonable balance between activities 
that accomplish organ i za t i ona 1 1 y defined objectives and 
those addressed to h i gh- poten t i a 1 exploratory projects? 

C. Will the balance between research and development be 
in keeping with the mission of the institution and the 
state of knowledge in the field? 

VI. Research Activities 

A. Are research objectives clearly specified? 

B. Are research objectives relevant both to the 
development objectives and to the overall goals of the 
i ns t i tu t i on? 

C. Are the proposed research designs scientifically 
adequate? 

D. Will the institution keep itself apprised of past and 
current research? 

VII. Development Activities 

A. Are development objectives clearly specified? 

B. Are development objectives relevant both to research 
objectives and to overall goals? 

C. Is the development process adequate? 

D. Are the procedures and timetable for achieving the 
objectives realistic? 

E. Will necessary, specialized development capabilities not 
available within the institution be acquired? 

V 1 1 1 . Re 1 a t i ons with Other Instituti ons , Agencies, and 
Organizations 

A. Will any collaborative relationships with outside 
organizations enhance the work of the proposed 
instituti on ? 

B. Will services required from other agencies be provided? 
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Schedu 1 e 



A. Is the timetable for carrying out the program reasonable? 

B. Do the points identified for recycling of activity come 
early enough to correct unsatisfactory outcomes at 
reasonable cost? 

C. Does the schedule provide genuine possibilities for 
accelerating or decelerating the tempo in case of 
changes in personnel or financial support? 

Facilities 



A. Are the equipment, materials, and physical facilities 
to be used in carrying out the program adequate? 



B. 



Fi 

A, 



B. 



C. 



Are they appropriate, sufficient, and economical for the 
work? 

nanc i ng 

Is the budget for the institution caref u’ 1 y prepared? 

1 - Is it comp 1 ete? 

2. Is it directly related to the proposed work as well 
as suf f i c i en t? 

3. Is it limited to what is required? 

Is the fiscal system complete and carefully devised? 

Are other i esources available to support the planning 
effort? 



Communications 

A. Are the clients and audiences of the institution 
likely to become aware of its significance and develop 
interest in its potential? 

B. Will the proposed plan lead to an increased 
constituency for research and development activites in 
educa t i on? 
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Dec i s i on 4 



U. S. Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 



Guidelines for Mature Status Application 

f o r 

Developing Research and Development Institutions 
Receiving Institutional Support from U. S. Office of Education 

Research and development institutions preparing for a terminal 
review at the end of their period of core support as developing 
institutions must prepare documents in accordance. wi th the 
instructions below. Those instructions contain the criteria on 
which the maturity of developing institutions will be judged; thus all 
information requested must be supplied. 





The major sections of the documents will 


be evaluated 


accord i 


ing to the following system of weighting. 






Sect i on 


We i gh t 


1 . 


Prog ram 


5 


1 1 . 


Management 


5 


til. 


Pe rsonne 1 


3 


IV. 


Planning and Evaluation System 


3 


V. 


Relations with Other Institutions, 
Agencies, and Organizations - - 


2 


VI . 


Fac i 1 i t i es 


2 


VII. 


F i nanc i ng 


2 


VIII. 


Commun i ca t i on 


1 
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I . Program 

A. Present the configuration of mi ss ion/program/components/ 
products so that both their completeness and their 
internal consistency will be apparent, 

1. Demonstrate that the products of the program are 
necessary for the institution to achieve its 
mission. 

2. Demonstrate that programs are derived from the mission, 
that components constitute coherent programs, and that 
components will produce the anticipated products. 

B. Give evidence to prove that the program plan, which the 
institution was funded to carry out, has been brought into 
f u 1 1 ope ration. 

C. Supply evidence tc show that program components have been 
achieving intermediate program objectives. 

D. Describe any refinement, revision, or extension of program 
plans as a result of the experience in carrying out the 
program plan. Supply the data on which the modification 
decisions were based, 

E. Describe the efforts of the institution to secure additional 
funding beyond what the U. S. Office of Education has 
supplied and discuss the relationship of the activities 
proposed for outside funding to the mission of the 

i ns t i tu t i on . 

I I . Management 

A. Supply detailed evidence that the management has established 
and can make effective use of procedures for: 

1. Planning and evaluation 

2. Recruiting, assigning, training, evaluating, promoting, 
and compensating personnel 

3. Monitoring detailed operations within all programs 

4. Executing a plan as outlined and according to schedule 

5- Producing documents that are brief, clear, cohesive, 
and revea ling 
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6* Supplying techn i ca 1 —support services 'for the program 
s taf f 

a. Providing audio-visual design and prodLStion 
se rv i ces 

b. Providing da ta-p rocess i ng support that is compete- 
access i b 1 e , speedy, ana produces reports in 
convenient formats 

7. Arranging for outside consulting and subcontract serv ces 

8* Performing housekeeping functions 

3. Operating a fiscal system in which the elements 

correspond to program components and which contains 
adequate safeguards, maintains detailed records, and 
produces accurate reports 

B. Outline the governing structure of the institution, 
explaining the part staff members as well as any governing 
board or external advisory committees play. 

C. Provide an adm i n i s t ra t i ve organization chart for the 
institution and describe its relationship to the organization 
of the basic program. 

D. Describe how decision-making authority and responsibility 
are allocated within the institution. 



Pe r sonne 1 

A. List the names and provide biographical data for professional 
personnel who have been employed. 

B. Specify the roles of professional staff members at all levels, 
giving brief ^job descriptions for major positions. 

C. Explain how the qualifications of the staff are relevant to 
the program in operation. Demonstrate that staff 

training and experience are sufficient to operate the program 
as planned. 



D. 



Supply evidence that the staff members understand the 
program in operation, are in agreement with it, 
and are committed to carrying it out. 



Describe the sources that will provide a continuing supply 
of regular personnel as well as consultants. 
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IV- Planning and Evaluation System 

A. Describe the planning procedures used to project activities 

for the institution. 

1. Illustrate how these procedures have been used to 
identify and assess alternative activities for achieving 
a particular goal. 

2. Illustrate how reports received Jby the planning unit 
have caused it to modify program plans. 

B. Describe the evaluation mechanism used by the institution. 

1. Give examples of how internal or external program 
reviews have been conducted. 



2. Give examples 
eva 1 ua t i on un 


of 
i t . 


p rocess 


eva 1 ua t i on 


pe rf ormed 


by 


the 


3. Give examples 
eva 1 ua t i on un 


of 
i t . 


p roduct 


eva 1 ua t i on 


pe rf ormed 


by 


the 



4. Illustrate how reports of process or product evaluations 
have led to refined operations or improved products. 

V. Relations with Other Institutions, Agencies, and Organizations 

A. Describe any formal sponsorship from or affiliation with 
other institutions, agencies, or organizations. 

B. Describe any collaborative relationships with outside 
organ i za t i ons . 

C „ Describe the services purchased from outside agencies and 
explain their relationship to the requirements of the 
p rogram. 

VI. Facilities 

A. Describe the equipment, materials, and physical facilities 
being used in carrying out the program. 

B. Demonstrate that these are appropriate, sufficient, and 
economical for the work. 
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V 1 1 . F i nanc i ng 

A. Describe in full detail the fiscal system used to hold, 
disburse, account for, and report on funds. 

B. Explain in detail the relationship of the fiscal system 
to the institutional program and demonstrate its 
usefulness to program personnel as well as to management. 

C. Display evidence that the institution is willing and able 
to allocate all institutional expenses to programs and 
has a satisfactory method for accomplishing this. 

D. Describe funding sources In addition to the U. S. Office 
of Education that might be able to support specific 
programs in the future. 

V 1 1 1 . Communications 

A. Give evidence that the institution has been publicized in 
such a way that the nature of its work has become clear 
and is regarded by outsiders as significant. 

B. Demonstrate that the institution is increasing the 
constituency for research and development activities 
in education. 



O 

ERIC 
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Dec i s i on 4 



U. S. Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 



Guidelines for Judges 
of 

Mature Status Applications 
for 

Developing Research and Development Institutions 
Receiving Institutional Support from U. S. Office of Education 



Judges must assess applications for mature status according 
to their standing on the criteria contained in the following set of 
questions. The questions are organized to parallel the sequencing 
the applicants followed in writing their proposals. All questions 
must be answered. 

The various sections of the applications will be evaluated 
according to the following system of weighting. 



Sect i on 

1 . Program 
I I . Management 

III. Pe r sonne 1 

IV. Program Planning and Evaluation 

V. Relations with Other Institutions, 
Agencies, and Organizat ions 

VI. Facilities 

VII. Financing 

VI II. Communication 



Weight 

5 

5 

3 

3 

2 

2 

2 

1 
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I I . 



P rog ram 

A. Is the configuration of m i ss i on/p rogram/ components/p roducts 
both complete and internally consistent? 

1. Will the institution achieve its mission if it completes 
the projected products? 

2. Does the proposed configuration of parts constitute a 
consistent set of linked elements? 

B. Has the program plan that the institution was funded to 
carry out been brought into full operation? 

C. Are program components achieving intermediate program 
ob j ec t i ves? 

D. Is the program plan being refined, revised, or extended as 
a result of the experience in carrying i.t out? Have the 
modifications been based on data? 

E. Will the institution be able to secure additional funding 
beyond what the U. S. Office of Education supplies? Will 
these funds advance Its mission? 



Management 



A. 



Has the management established and will it maks effective 
use of procedures for: 

1. Planning and evaluation? 

2. Recruiting, assigning, training, evaluating, promoting, 
and compensating personnel? 

3. Monitoring detailed operations within all programs? 

4. Executing a plan as outlined and according to schedule? 

5. Producing documents that are brief, clear, revealing, 
and cohes i ve? 

6. Supplying techn i ca 1 - suppo r t services for the program 
staff? 

a. Providing aud i o- v i sua 1 design and production 
se rv i ces? 

b. Providing data-process i ng support that is competent, 
accessible, speedy, and produces reports in convenient 
f o rma ts? 
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7- Arranging for outside consulting and subcontract 
se rv i ces? 

8. Pe r form i ng house keep i ng functions? 

9. Operating a fiscal system in which the elements 
correspond to program components and which contains 
adequate safeguards, maintains detailed records, and 
produces accurate reports. 

B. Is the governing structure of the institution well designed, 
including the way that staff members as well as any 
governing board or external advisory committees are placed 
in it? 

C. Is the adm i n i ns t ra t i ve organization of the institution 
closely related to the organization of the basic program? 

D. Are decision-making authority and responsibility allocated 
effectively within the institution? 

III. Pe r sonne 1 

A. Do the competence and experience of the professional 
personnel who have been employed clearly predict success 
for the institution and its programs? 

B. Is the staffing plan adequate? 

C. Do the training and experience of the staff match the needs 
of the program in operation? 

D. Do the staff members understand the program in operation? 
Are they in agreement with it and committed to carrying it 
out? 

E. Is there an adequate pool of manpower for both regular 
personnel as well as consultants? 

IV. Planning and Evaluation System 

A. Are the planning procedures used to project activities for 
the institution adequate? 

1. Are these procedures used to identify and assess 
alternative activities for achieving a particular goal? 

2. Have the reports received by the planning units caused 
any modification of its plans? 
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B. is the evaluation mechanism used by the institution adequate? 

K Are the internal or external reviews conducted in an 
effective manner by qualified persons? 

2. Does the : nstitution evaluate its ongoing work? 

3. Does the institution determine the extent to which its 
objectives are being achieved? 

4. Does feedback from evaluations of ongoing work or 
products lead to refined operations or improved 
products? 

V. Relations with Other Institutions, Agencies, and 0 rgan i za t i ons 

A. Does any formal sponsorship from or affiliation with other 
institutions, agencies, or organizations contribute to the 
success of the institution? 

B. Do any collaborative re 1 a t i onsh i ps with outside organizations 
enhance the work of the institution? 

C. Are required services that must be purchased from outside 
agencies being provided? 

V I . Fac i 1 i t i es 

A. Are there adequate equipment, materials, and physical 
facilities to carry out the program? 

B. Are they being used effect ively and economically? 

VII. F i nanc i ng 

A. Is the fiscal system used to hold, disburse, account for, 
and report on funds adequate? 

B. Is the fiscal system closely related to the institutional 
program? Is it easy to use and helpful to program 
personne 1 ? 

C. Is the institution willing and able to allocate all 
institutional expenses to programs and does it have a 
satisfactory method for accomplishing this? 

D. Are there sources in addition to the U. S. Office of 
Education that might be able to assist in f und i ng spec i f i c 
programs in the future? 
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VIII. Communications 

A. Do L, u .e clients and audiences of the institution understand 
clearly the nature of its work and regard it as 

s i gn i f i cant? 

B. Is the work of the institution increasing the constituency 
for research and development activities in education? 
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Decision 5 



U. S. Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 



Guidelines for Program Development Grant Applications 

f rom 

Mature Research and Development Institutions in Education 

Only mature research and development institutions are 
invited to apply for Program Development Grants. Institutions that 
have not been designated as "mature" by the Division of Research and 
Development Resources may request a maturity review. 

Applications must contain the information called for in the 
instructions below. The instructions contain the criteria on which 
proposals will be judged; thus all information requested must be 
suppl ied. 

The various sections of the proposal will be evaluated 
according to the following system of weighting. 



Section Weight 



1 . 


Problem Area 


5 


1 1 , 


Work Plan 


3 


1 1 . 


Pe rsonne 1 


5 


IV. 


F i nanc i ng 


2 




157 

±64 



158 



1 . Prob 1 em Area 

A, Clearly specify the problems, needs, or opport i n i t i es that 
the proposed program will address. 

B. State the objectives of the planning grant in operational 
terms. 

C * Describe the pupil populations, demographic areas, and 

economic settings that are the targets of the proposed work. 

D. Discuss the theoretical framework or rationale for the 
proposed work; or supply a review of relevant literature; 
or provide background data; or describe the discrepancy 
between current conditions and desired conditions; or cite 
special cha racte r i s t i cs of the geographic area; or otherwise 
support the view that the objectives of the planning grant 
need to be accomplished. 

E. Indicate whether the outcomes of the program will be 
national as well as regional in eventual impact. 

F. Explain how research and development activities will 

sat i sf actori ly deal with the problems, needs, or oppor t i n i t i es 
identified. 

G. Explain why the problem requires long-range programmatic 
research and development in an institutional setting and 
cannot be handled satisfactorily through the support of 
single p roj ects . 

H. Describe the relationship between the proposed work and what 
has been accomplished previously or is under way elsewhere. 

I* Relate the problem area to priorities in education. 

1. Describe the relationship to educational priorities 
recently expressed by the President; the Department of 
Health, Education, and Welfare; the U. S. Office of 
Education; the National Center for Educational Research 
and Development; or the Division of Research and 
Development Resources. 

2. Describe the relationship to educational priorities 
expressed by other leading individuals, organizations, 
and agenc i es ■ 
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3. In the absence of any demonstrable relationship to 

priorities expressed by government officials or other 
national figures, develop the argument that the selected 
problem area is nonetheless significant and merits 
support. 

A. Discuss the likelihood that the work will continue to 
be significant in the future, even if priorities shift. 

5. Discuss the significance in social and economic terms, 
as well as from an educational point of view. 

II. Work PI an 

A. Specify in precise terms the scope of work to be performed. 

B. List the critical questions that must be answered to achieve 
the objectives of the planning grant. 

C. Identify the formal or informal planning techniques to be 
used . 

D. Delineate the procedures for using evaluation techniques 
to assess and to guide the planning process. 

E. Explain how the planners will divide the planning tasks 
among themselves 

1. Explain the method for allocating tasks. 

2. Demonstrate that there will be a match between tasks 
assigned and competence to do them. 

3. Display a schedule for the use of the planning personnel 
showing their availability to carry out tasks in a 
planned series and to finish the work on time. 

F. Supply a complete schedule for the planning task. 

1. Supply starting points, milestone points, and ending 
points for major segments of the work. 

2. Justify the schedule, given the scope-of-work statement. 

III. Personnel 

A. Identify the person who will lead the planning effort. 

1. Describe his competency for this task. 

2. Indicate the amount of time he will devote to the work 
and the degree of responsibility he will accept. 
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B. Describe the organizational structure and the work pattern 

to be used by the planning group. 

1. Explain the use of teams, committees, or task groups, 
indicating how they will be led and interrelated. 

2. Indicate the amount of time and degree of involvement 
for each member of the planning group. 

C. Characterize the personnel. 

1. Supply data to show that they have both competency and 
relevant experience. 

2. Discuss the kind and number of personnel needed to 
carry out each task. Demonstrate that a critical mass 
of talent is available. 

D. List the classifications of personnel who will assist with 

the planning. 

1* Show that contributions from persons in the substantive 
disciplines will be made wherever relevant. 

2. Show that talent from technical fields such as planning, 
evaluation, and data processing is available. 

3. Show that sufficient specialty in both research and 
development is present within the planning group, 

4. Show that the ultimate clients of the institution, 
such as school systems, are represented among the 
p 1 anne rs - 

5. Show that key audiences for the work of the institution, 
such as members of state legislatures or key public 
figures, are involved in the planning. 

IV. Financing 

A. Supply a detailed budget for the planning effort* 

1. Specify the costs of personnel, equipment and materials, 
special services, travel, communications, and space 
rental . 

2. Demonstrate that the proposed budget is sufficient to 
carry out the scope of work. 

3. Identify those points at which the budget might be 
reduced without impeding the planning. 
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Dec i s i on 5 



U. 5. Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 



Guidelines for Judges 
of 

Applications for Program Development Grants 

f rom 

Mature Research and Development Institutions in Education 

Judges will assess applications for Program Development 
Grants according to their standing on the criteria contained in the 
following set of questions. The questions are Organized to parallel 
the sequencing the applicants followed in writing their proposals. 
All questions must be answered. 

The various sections of the applications will be evaluated 
according to the following system of weighting. 

Sect i on Weight 



1 . 


Problem Area 


1 1 . 


Work Plan 


III. 


Pe rsonne 1 


IV. 


F i nanc i ng 
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Prob 1 em Area 



I . 



A. 



Do clearly identified problems, needs, or opportunities 
guide the institution in formulating the proposed program? 



B. Are the objectives of the planning grant stated in operational 
te rms? 

C. Are significant pupil populations, demographic areas, and 
economic settings singled out as targets of the proposed 
p rogram? 



D. Is there a sound theoretical framework or rationale for the 
proposed work; or a persuasive review of relevant literature; 
or relevant and important background data; or a convincing 
accounting of a discrepancy between current conditions and 
desired conditions; o; special cha racter i s t ? cs of the 
geographic area that demand attention; or other evidence 
that the objectives of the planning grant need to be accomplished? 



E. Can the outcomes of the program be national as well as 
regional in eventual impact? 



F. Can the research and development activities deal sat i sf actor i 1 y 
with the problems, needs, or opportunities identified? 



G. Does the problem require long-range programmatic research 
and development in an institutional setting rather than 
the support of single projects? 



H. Does the proposed work build on without duplicating what 
has previously been accomplished or is underway elsewhere? 

I. Is the mission or problem area related to priorities in 
educat i on? 



1. Is it related to educational priorities recently 
expressed by the President; the Department of Health, 
Education, and Welfare; the U.S, Office of Education; 
the National Center for Educational Research and 
Development; or the Division of Research and Development 
Resources? 

2. Is it related to educational priorities expressed by 
other leading individuals, organ i zat i ons , and agencies? 

3. In the absence of any demonstrable relationship to 
priorities expressed by government officials or other 
national figures, does it develop the argument that 
the selected problem area is nonetheless significant 
and merits support? 
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4. Will the work continue to be significant in the 
future, even if priorities shift? 

5. Will achieving the proposed mission be significant 
in social and economic terms, as well as from an 
educational point of view? 

II. Work P 1 an 

A. Is the scope of work specified in precise terms? 

B. Arc the questions to be answered critical to achieving 
the objectives of the planning grant? 

C. Wilt the proposed formal or informal planning techniques 
lead to successful completion of the scope of work? 

D. Will the proposed evaluation techniques effectively assess 
and guide the planning process? 

E. Will the best possible division of tasks among the planning 
personnel be effected? 

1. Will the method for allocating the tasks work out in 
actual practice? 

2- Is there a close match between tasks assigned 
and competence to do them? 

3. Will the planning personnel carry out their tasks in 
a planned series and finish the work on time? 

F. Is the schedule for the planning tasks balanced and complete? 

1. Does it supply precise starting points, milestone 
points, and ending points for major segments of the work? 

2. Is the schedule closely related to the scope-of- 
work statement? 

III. Personnel 

A. Will the person who will lead the planning effort be effective? 

1. I s he competent for this task? 

2. Will he give enough time to and take enough responsibility 
for the work? 

B. Are the organizational structure and the work pattern to be 
use^ by the planning group appropriate for the task? 
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IV. 



1 . 



Will the best possible use be made of the proposed 
work groups? 



2 . 



Will the amount of time and degree of involvement for 
each member of the planning group be adequate to the task? 



C. Are the personnel properly selected for the work? 

1. Will the personal competence and relevant experience of 
each lead to successful completion of the planning 
effort? 



2. Will a critical mass of specified talents be available? 



D. Will an adequate range of personnel assist with the planning? 

1. Will contributions from persons in the substantive 
disciplines be made wherever relevant? 

2. Will talent from technical fields such as planning, 
evaluation, and data processing be available? 

3. Will sufficient specialty in both research and development 
be present within the planning group? 

4. Will the ultimate clients of the institution, such as 
school systems, be represented among the planners? 

5- Will key audiences for the work of the institution, 
such as members of state legislatures or key public 
figures, be involved in the planning? 



Fi nanci ng 



Is 


the 


1 . 


Is 


2. 


1 s 


s. 


Is 



budget for the planning effort carefully prepared? 
it complete? 
it sufficient? 

it limited to what is required? 
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Dec i s T on 6 



IL S. Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 



Guidelines for Program Plan 
for 

Mature Research and Development Institutions in Education 



Only mature research and development institutions are 
invited to apply for program support. Institutions that have not been 
designated as "mature" by the Division of Research and Development 
Resources may request a maturity review. 

Applications must contain the information called for in the 
instructions below. The instructions contain the criteria on which 
proposals will be judged; thus all information requested must be 
suppl i ed. 



The various sections of the proposal will be evaluated 
according to the following system of weighting. 



Sect i on 



We ? qht 



Relation of Program to Previous Work 
of the Institution 



II. P rob 1 em 

III. Program 

IV. Personnel 

V. Planning and Evaluation System 

VI. Relations with Other Institutions, 
Agencies, and Organizations 

VII. Schedule 
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Sect i on 


Wei ght 


VI 1 1 . 


Facilities 


3 


IX. 


Fi nanci ng 


3 


X. 


Commun i cat ? on 


2 


XI . 


Product Dissemination Plan 


1 


XI 1 . 


Contribution to Training 


1 
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j. Relation of Program to Previous Work of the Institution 

A. Describe the previous work of the institution in the 
proposed program area or in a closely related program area. 

B. State explicitly how this previous institutional 
experience can be used to enhance the performance of the 
the proposed work. 

C. Supply specific information about the schedule and 
costs of related previous work that will authenticate 
the schedule and costs for the proposed work. 

1 1 . Prob 1 em 

A. Clearly specify the problem, need, or opportunity that 
the proposed program will address. 

B. Del ineate the problem area and explain it in terms 
specific enough to provide focus yet general enough to 
justify long-range programmatic work. 

C. Describe the pupil populations, demographic areas, and 
economic settings that are the targets of the proposed work. 

D. Authenticate the problem, need, or opportunity by presenting 
a theoretical framework or rationale for the proposed work; 
or supply a thorough review of relevant literature; or 
delineate the discrepancy between current conditions and the 
desired conditions; or cite special characteristics of the 
goegraphic area or otherwise establish conclusively that the 
proposed program should be carried out. 

E. Indicate whether the outcomes of the program will be 
national as well as regional in eventual impact. 

F. Justify the assertion that research and development 
activities will satisfactorily deal with the problem, 
need, or opportunity identified. 

G. Explain why the problem requires long-range programmatic 
research and development in an institutional setting and 
cannot be handled satisfactorily through the support of 

s ingle proj ect s . 

H. Describe the relationship between the proposed work and 
what has been accomplished previously or is under way 

e 1 s ewhere . 

I. Explain how the mission is in keeping with generally accepted 
educational values. 



O 
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J. Relate the problem area to priorities In education. 

1. Describe the relationship to educational priorities 
recently expressed by the President; the Department 
of Health, Education, and Welfare; the U. S. Office 
of Educational; the National Center for Educational 
Research and Development; or the Division of Research 
and Development Resources. 

2. Describe the relationship to educational priorities 
expressed by other leading individuals, organizations, 
and agencies. 

3. In the absence of any demonstrable relationship to 
priorities expressed by government officials or 
other national figures, develop the argument that 
the selected problem area is nonetheless significant 
arid merits support. 

4. Discuss the likelihood that the work wi 1 1 continue to 

be significant in the future, even if national priorities 
shift. 

5. Discuss the significance in social and economic terms, 
as well as from an educational point of view. 



I 1 I # P rogram 

A. Specify defensible, cler-, important objectives for the 

program. 

1. Supply evidence to justify selection of the objectives 
and to demonstrate that they were chosen reasonably 
from among a set of alternatives. 

2. State the objectives in operational terms, clearly 
specifying what the terminal product of each objective 
will be. For example. Is the product to be new knowledge, 
some new forrr> of educational practice, or new instructional 
mater i al s? 

3. indicate the pupil populations, demographic areas r and 
economic settings to be affected by achievement of 
the objectives. 

4. Demonstrate that the objectives strike a favorable 
balance between addressing critical needs and using 
available opportunities for meeting needs. 

5. State whether achievement of the objectives is expected 
to make an incremental improvement, a sharp modification, 
or a complete change in the schools. 
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B. Specify defensible, clear, and relevant procedures for 
achieving the objectives* 

1. Supply evidence to justify the selection of the 
procedures and to demonstrate that the choice among 
the available alternatives was reasonable. 

2 . State the procedures clearly in operational terms. 

3. Show how the procedures are related directly to 
achieving the objectives. 

A. Describe the mechanisms for evaluating both the 

program processes and the products and demonstrate 
their adequacy for sel f- cor rect i on and redirection. 

I V. Personnel 

A. Specify the roles of staff members at all levels, giving 
brief job descriptions for major positions. 

B. Explain hdw the qualifications of the staff are relevant to 
carrying out the proposed work. Demonstrate how staff 
experiences are sufficient in variety and comprehensiveness. 

1. Show that contributions from persons in the 
substantive disciplines will be made wherever relevant. 

2 . Show that talent from technical fields such as 
planning, evaluation, and data processing is available. 

3. Show that sufficient specialty in both research and 
development is present within the staff. 

C. List the names and provide biographical data for the key 
personnel who have agreed to work In the program when it 
i s funded. 

D. Supply evidence that persons who have been engaged in the 
planning and will continue as permanent staff members 
understand the plan, are in agreement with it, and are 
committed to carrying it out. 

E. Describe the sources that will provide a continuing 
supply of regular personnel as well as consultants. 

V. Planning and Evaluation System 

A. Describe the procedures that will be used to monitor and 
assess activities at the operational level. 
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]. Describe the content and format of reports that will be 
supplied to program people to guide their work. 

2. Supply the schedule on which these reports will be 
p rov i ded . 

B. Describe the methods that will be used to determine 

the extent to which program objectives are being achieved. 

C. Give a complete description of the method that wi 1 1 be 
used to conduct milestone reviews at key points throughout 
the program. 

1. Describe the internal review procedure that will be 

used by program personnel or others within the institution. 

2. Describe the external review procedure that will be used 
Involving personnel from outside the Institution. 

(External milestone reviews are required for all funded 
p rog rams . ) 

3. Describe the format and schedule for transmitting 

the results of these milestone reviews to the vision 
of Research and Development Resources In the S. 

Office of Education. (The schedule must be in 
sufficient detail to allow close monitoring by the 
Office of Education.) 

D. Describe the nature of the final report that will be 
transmitted to DRDR at the conclusion of the program and 
the date for transmitting it. (Final reports are required 
for all funded programs.) 

VI. Relations with Other Institution, Agencies, and 0 rgan i zat i ons 

A. Describe any col 1 abo rat i ve relationships with schools, 
colleges, state departments of education, research and 
development organ i zat i ons , ESEA Title I and Title III 
projects,, or other federal programs. Explain hew these 
relationships will assist the program in achieving its 
ob j ect i ves . 

B. Indicate how specialized capabilities not available within 
the institution will be acquired, e.g., through 
consulting arrangements or subcontracts. 

C. Describe and, if possible, supply data on the attitudes 
held by others who might impede the progress of the 
program or ultimately interfere with its accomplishments. 
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Schedul e 

A. Specify the starting points, milestone points, and 
ending points for all programs end all program components. 

B. Identify the points at which recycling of activities can 
correct unsat i sf actory outcomes. 

C. Identify the possibilities for accelerating or decelerating 
the tempo in case of changes in personnel or financial 
support . 

Faci 1 i t i es 

A. Describe the equipment, materials, and physical facilities 
to be used in carrying out the program. 

S. Demonstrate that these are appropriate, sufficient, 
and economical for the work. 

F i nanci rig 

A. Supply a detailed budget. 

1. Specify the costs of personnel, equipment and 
materials, special services, travel, communications, 
and space rental . 

2. Demonstrate that the proposed budget is directly 
related to as well as sufficient to carry out the 
proposed work. 

3. identify those points in the budget that might be 
reduced without impeding the process of the program. 

B. Describe the fiscal system to hold, disburse, and 
account for funds. 

C. Describe other resources available to support the program. 
Conimun ? cat i on s 

A. Describe a plan for publicizing the program that will make 
the nature of the work clear, demonstrate its significance, 
and arouse interest in its potential. 

B. Describe how this can be done so that it will increase the 
contituency for research and development activities in 
educat i on . 
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X 1 # Product [)i ssemi nat I on Plan 

A. Describe the plan for ultimately disseminating the 
products of the program should It p ~ove successful. 

B. List the possible funding sources that might support 
the d! ssem? nat Ion effort. 

XI!. Contribution to Training 

A. Indicate the number of professional personnel who will 
receive training in research and development if the 
program is funded. 

B. Explain whether the training will be supplied in formal, 
scheduled sessions or will be supplied as informal, on- 
the-job training. 
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Dec I s I on 6 



U. S. Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 



Guidelines for Judges 
of 

Program Plans 
f rom 

Mature Research and Development Institutions in Education 



Judges must assess the program plans according to their 
standing on the criteria contained in the following set of questions. 
The questions are organized to parallel the sequencing the planners 
followed in writing their program plans. All questions must be 
answe red . 



The various sections of the proposal will be evaluated 
according to the following system of weighting. 



Sect i on 



Wei qht 



l # Relation of Program to Previous Work 3 

of the Institution 



II. 


Problem 




III. 


Program 




IV. 


Pe rsonnel 




V. 


Pi ann i ng 


and Evaluat 


VI . 


Rel at I ons 
Agenc i es , 


with Other 
and Organi 


VII. 


Schedu 1 e 





5 

5 

5 

ion System 3 

I nst i tut i ons, 2 

zat ? ons 

5 
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Sect i on 


Wei 


VIII. 


Faci 1 i t i es 


3 


IX. 


F i nanci ng 


3 


X. 


Commun T cation 


2 


XI . 


Product Dissemination Plan 


1 


XII. 


Contribution to Training 


1 
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! . Relation of Program to Previous Work of the Institution 

A. Does the previous work of the institution in the proposed 
program area or in a closely related program area indicate 
that it can succeed with the proposed program? 

B. Can this previous institutional experience be used specifically 
to enhance the performance o f the proposed work? 

i 

C s Does the previous experience of the institution authenticate 
the schedule and costs for the proposed work? 

II. P rob 1 em 

A. Do clearly identified problems, needs, or opportunities guide 
the institution in setting its mission? 

B. Is the problem area outlined and the mission explained in terms 
specific enough to provide focus yet general enough to justify 
long-range programmatic work? 

C. Are significant pupil populations, demographic areas, and 
economic settings singled out as targets for the proposed work? 

D. Is there a sound theoretical framework or rationale for the 
proposed work; or a persuasive review of relevant literature; 
or relevant and important background data; or a convincing 
accounting of a discrepancy between current conditions and 
desired conditions; or special characteristics of the geographic 
area that demand attention ;or other evidence that the proposed 
mission should be performed? 

E. Will the outcomes of the work be national as well as regional in 
eventual impact? 

F„ Can research and development activities deal sa t i s f ac to r i 1 y with 
the problems, needs, or opportunities identified? 

G. Does the problem require long-range programmatic research and 
development in an i ns t i tu t i ona 1 setting, or can it be solved 
through the support of single projects? 

H. Will the proposed work build app rop r i a te 1 y on what has been 
accomplished previously or is underway elsewhere? 

I. Is the mission in keeping with generally accepted educational va 1 

J. Is Lhe mission or problem area related to priorities in education 

1. Is it related to educational priorities recently expressed 
by the President; the Department of Health, Education, and 
Welfare; the U.S. Office of Education, 
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the National Center for Educational Research and 
Development; or the Division of Research and 
Development Resources? 

2. Is it related to educational priorities expressed 
by other leading individuals, organizations, and 
agenci es? 

3. In the absence of any demonstrable relationship to 
priorities expressed by government officials or 
other national figures, is the selected problem area 
nonetheless significant enough to merit support? 

4. Will the work continue to be significant even if 
national priorities shift? 

5. Will achieving the proposed mission be significant 
in social and economic terms, as wel 1 as from an 
educational point of view? 



Ilf. P rogram 

A. Are the objectives defensible, clear, and important? 

1. Are the selected objectives the best choice from among 
the range of possible objectives? 

2. Is the terminal product of each objective clearly 
speci f i ed? 

3. Are the targets of the p rogram--pup i 1 populations, 
demographic areas, and economic settings — clearly 
speci f i ed? 

4. Do the objectives strike a favorable balance between 
addressing critical needs and using available 
opportunities for meeting needs? 

5. Is the expected outcome — either an incremental 
improvement, a sharp modification, or a complete 
change In the school s--the best choice of the three 
a 1 ternat i ves , given the nature of the problem. 

B. Are the procedures for achieving the objectives defensible, 

clear, and relevant? 

1. Are the selected procedures the best choice from among 
the range of possible alternatives? 

2. Are the procedures clearly stated in operational terms? 
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3- Are the procedures clearly related to achieving the 
ob j ect i ves? 

k w Will the selected evaluating mechanisms lead to 

effective sel f- co r rect i on and redirection of both the 
program processes and the products? 

I V. Personnel 

A. Is the staffing plan adequate? 

B. Do the qual i f i cations of the staff match the needs of 
the program? 

1. Are contributions from persons in the substantive 
discipl ines made wherever relevant? 

2. Ss talent from technical fields such as planning, 
evaluation, and data processing used when needed? 

3. ! s sufficient specialty in both research and development 
present within the staff? 

C. Does the previous experience of the key personnel who have 
agreed to work in the program when it is funded lead to 
confidence that the program will succeed? 

D. Do the persons engaged in the planning, who will continue 
as permanent staff members, understand the plan? Are 

they in agreement with it and committed to carrying it out? 

E. Is there an adequate pool of manpower for regular personnel 
appointments as well as for consultants? 

V. Planning and Evaluation System 

A. Will the procedures, which will be used to monitor and assess 
activites at the operational leve 7 ,help the program personnel? 

1. Will the content of the reports, which will be supplied to 
program people actually help to guide their work? 

2. Will the schedule, on which these reports will be supplied, 
be useful to program personnel? 

B. Are the methods satisfactory that will be used to determine 
the extent to which program objectives are being achieved? 

C. Is the method adequate that will be used to conduct milestone 
reviews at key points throughout the program? 
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3 Is the internal review procedure adequate that will 
)e used by program personnel or others within the 
institution? 

2. Is the external review procedure adequate that will oe 
used involving personnel from outside the institution? 

3- Do the format and schedule for transmitting the results of 
these milestone reviews to the Division of Research and 
Development Resources in the U S Office of Education 
appear reasonable? (The schedule must be in sufficient 
detail to allow close monitoring by the Office of 
Edu ca t i on . ) 



D. Are the contents and date for submitting the final report 
sufficiently specified? 



VI. Relations with Other Institutions. Agencies, and 0 r ga n i za t i ons 

A. Will any co 1 1 abo ra t i ve relationships with outside organizations 
assist the program in achieving its objectives? 

B. Will specialized capabilities not available within the 
institution be acquired? 

C. Are outsiders likely to impede the progress of the program 
or ultimately interfere with its accomplishments? 

VII- Schedu I e 

A. Are the starting points, milestone points, and ending points 
for all programs and all program components reasonable? 

B. Are the points at which recycling of activity can correct 
unsat isfactory outcomes early enough to allow prompt 
correction of troubles that ocurr? 

C. Do the possibilities for accelerating or decelerating the 
tempo in case of changes in personnel or financial support 
allow substantial flexibility? 

V 1 I I „ Facilities 

A. Are the equipment, materials, and physical facilities to be 
used in carrying out the program adequate? 

B. Are they appropriate, sufficient, and economical for 
tne work? 
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IX. F i nanc i ng 



A, 


1 s 


the 


budget carefully prepared? 






1 . 


i s 


i t comp 1 e te? 






2. 


1 s 


i t suf f i cl ent? 






3. 


1 s 


it limited to what is required? 




B. 


1 s 


the 


fiscal system complete and carefully 


dev i sed ? 


C . 


Are 


other resources available to support the 


p I ann 1 ng 



X. Communications 

A. Are the clients and audiences of the institution likely to 
become aware of its significance and develop interest in its 
po te n t i a 1 ? 

B. V/ i 1 1 the proposed plan lead to an Increased constituency for 
research and development activities in education? 



XI. Product Dissemination Plan 

A. Is the plan ter ultimately disseminating the products of the 
program reasonable? 

B. Are there possible funding sources that might support the 
dissemination effort? 



XII. Contribution to Training 

A. Will a reasonable number of professional personnel receive 
training in research and development if the program is funded? 

B. Is the training to be supplied in formal, scheduled sessions or 
only as informal, on-the-job training? 
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Dec i s i on 7 



U. S. Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 



Milestone Review 

DRDR Evaluation Unit Responsibilities 



Program milestone reviews will be conducted by all mature 
research and development institutions receiving DRDR program support. 
All such institutions, when they applied for program support, filed 
milestone review plans with DRDR that contained a schedule for 
conducting the reviews and the date and format of the reviews for 
sending to DRDR. 

Members of the DRDR evaluation staff wi 1 1 maintain a 
calendar of dates for receiving milestone reviews and will follow-up 
with any institution that fails to supply the review on the schedule 
p rom i sed . 



When the milestone reviews are received by DRDR, the 
evaluation staff will examine them to see whether milestones were 
reached on time at a satisfactory level of quality. The findings 
of the milestone reviews will be summarized and reported to DRDR 
decision-makers, with an intervention review recommended if indicated 
by the findings. Moreover, they will be filed with the records of 
the research and development institution and used by the DRDR 
evaluation staff to inform DRDR decision-makers and program monitors 
when periodic funding decisions are being made. 
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Dec i s i on B 



U. S. Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 



Intervention Review 
DRDR Evaluation Unit Responsibilities 



Ocassionally DRDR- supported research and development 
work in mature institutions will not proceed as expected. For 
example, milestone reviews will indicate unsatisfactory performance, 
research or development breakthroughs will occur that could have 
a major effect on the future of the program, key personnel will 
l eave the institution, or reports from cooperating institutions 
will indicate that the work is not proceeding according to plan. 

When such occasions arise, intervention . eviews may be requested 
by USOE. 



Inasmuch as each intervention review is likely to call for 
ad hoc arrangements by the DRDR evaluation unit, general instructions 
cannot be written. However, it is likely that the guidelines and 
instruments used at other major decision points could be selected 
in some combination for an intervention review. And it may be 
necessary to eliminate certain sections from the guidelines and 
instruments or to develop supplementary instructions for reviewers. 
This will be the responsibility of the DRDR evaluation staff and 
any necessary outside experts. 
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Dec i s i on 9 



U. S. Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 



Program Outcome Review 
DRDR Evaluation Unit Responsibilities 



A detailed plan for program outcome evaluation is being 
prepared for DRDR under a contract issued by the Office of Program 
Planning and Evaluation. 

The exact responsibilities of the DRDR evaluation unit t 
assessing program outcomes presumably w i 1 1 be determined jointly by 
NCERD and the Office of Program Planning and Evaluation. The latter is 
planning to retain res pons f b i 1 i ty for evaluati ng the impact of Office 
of Education proarams . Thus, the outcomes of DRDR- supper ted r £ d 
programs possibly, will be evaluated in whole or in part Ly OFPE. 

Regardless of the outcomes of the above transact ions , DRDR 
will need information about program outcomes to guide its future 
funding and dissemination decisions. Therefore, when the last 
milestone has been reached in any given program, the DRDR evaluation 
unit will employ three independent experts to examine the program’s 
products and reports of final tests run on them. Then the experts 
will file independent reports giving ORDR their judgments as to the 
product's quality. 
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Dec i S i on 1 0 



U. S. Office of Education 

National Center for Educational Research and Development 
Division of Research and Development Resources 



Budget and Contract Review 
DRDR Evaluation Unit Responsibilities 



DRDR makes annual funding decisions for programs already in 
operation (tnose that pass decision point 6) and contracts for a 
specified scope of work in each program for the year ahead.. DRDR's 
policy insures that programs will be funded during satisfactory 
performance until a prescheduled termination point is reached. 

The DRDR evaluation staff usual ly wi 1 1 not be involved in these 
annual funding decisions. The evaluation unit's responsibilities to 
DRDR decision-makers for such occasions will be discharged partly by 
monitoring the milestone reviews conducted by mature institutions 
receiving continuing program support (decision point 7). Inasmuch 
as these milestone reviews occur at irregular times throughout the 
year for all the mature research and development institutions being 
supported by DRDR, the monitoring work of the evaluation unit cannot 
be scheduled to coincide with annual funding decisions. However, it 
is the responsibility of the evaluation unit to draw information 
about milestone reviews from the files and to supply it in an orderly 
fashion to DRDR decision-makers at the time of annual funding decisions. 
This is important because information on the success or failure of 
programs in reaching their milestones will influence funding decisions 
for the year ahead. Similarly, any information gathered by the DRDR 
evaluation staff during intervention reviews will be filed and drawn 
out and summarized at the time of annua* funding decisions. The 
same thing will occur to information that becomes available to the 
DRDR evaluation staff during the assessment of program outcomes - 
whether it was conducted by the research and development institution, 
the DRDR evaluation staff, or by the OPPE staff. 

In short, the DRDR evaluation staff, at the time of annual funding 
decisions (decision point 10) will d aw from the files any relevant 
data on the performance of the mature research and development institutions 
receiving program support and summarize this information for DRDR 
decision-makers. 
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Appendix B 



Re 1 a t i onsh i 
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Between Selected Evaluation Factors 



and 

Suppo r t- Po 1 i cy Decisions 
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Re 1 a 



(i) 



( 2 ) 



( 3 ) 



(V 



Should the request 
for a planning grant 
f o r a n ew ? ns t i tu- 
t i on be funded? 



Should a given 
proposal to start 
a new institution 
be funded? 



Is the basic 
program plan of 
a g i ven insti- 
tution satisfac 
tory ? 



1 s the institution 
capability to man- 
age its basic p ro- 
gram p-an satisfac- 
tory? 



Should 3 
deve 1 oprf 
for a ne 
be funde 




always a p p 1 i cab 1 e 

applicable only when planning grant reports provide insufficient data 
may or may rot be requ i red dependent upon overall funding constraints 
and the availability of pertinent information already on file 
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Relationship Between Selected Eval uation -actors and Suppo r t- Po 1 i cy Decisions 



(5) 



( 6 ) 



t i on 

nan- Should a given 

pro- development grant 

sfac- for a new program 

be funded? 



EVALUATION FACTORS 



Should the Office 
of Education commit 
itself to support a 
given proposed new 
program plan? 



A re s i gn i 
mi lestone; 
program pi 
i ng reachc 
t i me and v. 
i sf actory 




D ECISION CONSEQUENCES 
Allocation of Funds 
Cert i f i ca t i on as Mature 
Termination of Support 

CRITERIA FOR JUDGING 
New I ns t i tut i on 
New Program 
Existing Institution 
Existing Program 



Institution 




SOURCES OF DATA 
Proposa 1 s 

Progress and Final Reports 



FOCUS OF EVALUATION 

Single Program or Institution 

Multiple Programs or Institutions 



PROCEDURE FOR JUDGMENT 
Reading by 3 Judges 
Reading by 5 Judges 
Site V : sit by 3 Judges 
Site Vi >it by 5 Judges 



TYPE OF ANALYSIS 
Rank Order 
I nd i vi dua 1 



RESPONSIBILITY FOR ANALYSIS 
External Agency and DRDR Staff 
DRDR Staff Only 

EXTERNAL AUDIT 

Required on a sampling basis 



ifl 
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OPS 





(7) 


(8) 


(3) 






Do current circum- 






Are s i gn i f i cant 


stances of the 




;e 


mi 1 "ones i n the 


programs of institu- 




';171 i t 


program plans be- 


t i ons requ i re 




r t a 


ing reached, on 


red i rec t ion or 


0 i d the program 


~sew 


t i me and with sat- 


changes of Office of 


reach its 




isfactory quality? 


Educatlor s jpport? 


obj ect t ves? 




(10) 

With given funds 
each year, how 
should any given 
program be adjus- 
ted, stretched, 
or compressed? 















Chart showing 

R elationship Between Selected Evaluation Factors 
and Support-Policy Decisions 



194 






